This interview was conducted by Serena, an Instagram engineering manager.

How did you become an engineer?

When I was in college, the field of Data Science did not yet exist. I studied Statistics & Biometry in undergrad. Later on, I went to graduate school for Engineering.

What was your first coding language?

I used Java when I took my first computer science class, but most of my math and statistics classes required SAS, SPSS, and MATLAB.

What do you listen to while you work?

Believe it or not, I don’t listen to music at all while working. I think it’s distracting. Plus, these days I’m not at my desk long enough to listen to music anyway.

What do you do when you get stuck on a problem?

I like to talk it out with people. Discussing the problem out loud helps me think about the problem differently, and it helps to hear from other people who either have encountered a similar problem before or may look at things from a different perspective.

What do you do when you feel a lot of pressure?

I usually like to take a break. If I have time, I’ll take a walk around the NY building’s floors or our office’s amazing rooftop garden. If I don’t have time, I just take a beat — I take some deep breaths to clear my head, and that can be very helpful.

Please tell us about your favorite project at IG.

This is like being asked to pick your favorite child! My recent favorite is the private like counts test, a product exploration aimed at shifting the focus from the quantity of interactions to the quality of interactions on your feed posts. What I really love about it is that we are challenging the status quo. We are thinking differently about a part of the product that has been a vital part of the Instagram experience from the beginning.

What makes working at Instagram unique?

After two years here, I can confidently say that it’s our culture. One of our values is “people first” — meaning we’re always laser focused on how to create the best experience for our community. We approach every project through the lens of the people who use Instagram.

How would you describe the engineering culture at Instagram?

Very collaborative. We put the Instagram community first, and we work together to create the best products. Everyone here is so passionate about building amazing products for our community, and it’s so energizing to be a part of it.

What makes you excited about coming into work every day?

Working on a product that impacts the lives of billions of people throughout the world. It’s incredibly exciting. It’s also amazing to think about the potential impact we can have with every new idea and project we work on. It’s a huge responsibility, and one that we and I take very seriously.

Your favorite place to eat in the city?

I love sushi. Sushi of Gari is my favorite neighborhood sushi place in New York.

What is your favorite thing to eat at the office?

A smoothie from Frozen Palm, our smoothie bar in the New York office, which provides a lot of great (and healthy!) options.

What’s your favorite Instagram account?

@girlswhocode, which is an organization that I volunteer for. They work on (and post about) important research that’s relevant to women in tech, and their ultimate goal is to help young women find career paths in STEM. I like to follow along to see what programs they’re working on and to hear stories about some of the alumni.

Tell us about your happiest day at Instagram

It was my second Faceversary (second-year mark at Facebook) and my team pulled out all the stops with flowers, cupcakes, etc. My manager, Adam Mosseri, was out of the office, but Nam, our head of engineering, surprised me at a meeting by printing Adam’s face on a piece of paper. We took a selfie with the cutout of Adam’s face, pretending he was there and celebrating with us. It was hilarious, but also very special for me because it was a great way to reflect back on all that’s happened over the past couple of years here.

What is one of the best things you learned while working at Instagram

The importance of good communication and context sharing. We work in such a fast-paced environment that often we assume people have the context they need to do their jobs well. Sometimes, it’s important to slow down and take the time to communicate the bigger picture.

Coolest celebrity sighting at the office?

The cast of Crazy Rich Asians visited the office, and many people were excited to see them in real life. We welcomed Henry Golding, Michelle Yeoh, Awkwafina, Ken Jeong, and book author Kevin Kwan for a FB Live from our New York office where they answered fan questions, and @evachen went Live with the cast from her Instagram account for a Q&A ahead of the film’s release. If you haven’t seen the movie, you’re missing out :)

What does your desk setup look like?

It’s super clean. I am a neat freak and a minimalist too. I think a clean space helps me keep my focus. But it’s hard to keep a clean desk here because there’s always some new piece of swag that manages to make its way onto my desk.

What career advice would you like to give to female engineers who are early in their careers?

So many — but one I’d like to touch on here is the importance of networking. It’s easy to think that we need to just sit at our desks and do really good work, but that’s only one part of the equation.

It’s important to build relationships with people who are not in your immediate team, or company. Networks can open up opportunities for you in the future, and it’s also important to communicate your work with a broader audience. As women, I don’t think we advocate enough for ourselves, and we need to do a better job communicating our accomplishments.

Another is confidence. Women, especially those early in their careers, are often prone to self-doubt. It ends up being our biggest enemy as we second-guess ourselves too much. It prevents us from pursuing opportunities to be challenged and grow. How can we overcome it? For me, what really worked was the “fake it until you make it” mindset. Observe the behavior of your confident colleagues and try to mirror it. On this topic, I really recommend The Confidence Gap.

How do you achieve work-life balance? Do you believe in work-life balance at all? :)

I do! It’s all about setting boundaries and not apologizing for these boundaries. We need to be very diligent about where we draw that line. For me, this might mean that I leave the office by a certain time so I can have quality time with my kids in the evening, and for others it might be something else. The key is sticking to these boundaries and carving out the time that you need to achieve that balance.

Do you think work-life balance is harder to achieve as you become more and more senior in the company?

I don’t think so. The demand definitely becomes greater, but you become better at ruthless prioritization.

Most of the time, I don’t feel hindered by being a female engineer as I don’t have a problem speaking up or leaning in. But there are still times — for example, when I am pregnant — I worry that people will think less of me, and I worry whether I will still be able to meet my colleagues’ expectations. How can we overcome these thoughts?

It’s a really great question and one that goes back to the confidence piece I mentioned earlier. Creating a human being is something to be very, very proud of — it’s amazing!

We need to think about success in our lives in a broader way than just our work. You are building a family, you are a talented engineer, and all of these things are what make you you.

I find that concerns around performance during a pregnancy are often a manifestation of our own self-doubt. While you might not feel as good or sharp as you’re used to, others are probably unaware, and you are your own worst critic. Everyone has good days and bad days, when they’re feeling distracted or not performing their best. We need to be kind to ourselves and be confident in our abilities.

If you want to learn more about this work or are interested joining one of our engineering teams, please visit our careers page, follow us on Facebook or on Twitter.

Interview with Tamar Shapiro, Instagram’s Head of Analytics was originally published in Instagram Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Written by Amy Huyen, 2-time intern at Instagram.

On my first day of intern orientation last summer, I walked into Facebook wide-eyed and in awe of all the talent that surrounded me. I saw my peers networking with one another, discussing past internship experiences, and using technical jargon that I couldn’t understand. I felt like an impostor.

I was an inexperienced freshman who had only been coding for 10 months. I had no industry experience, was only versed in Python, and couldn’t even begin to comprehend what tech at a big company was like. Unlike most software engineering interns, I had been admitted into Facebook’s freshmen-sophomore Facebook University program (“FBU”). Instead of working directly on company products, FBU interns spend three weeks in an intensive mobile development bootcamp, then five weeks developing their own mobile apps in teams of three.

I should’ve been ecstatic to be surrounded by such brilliant individuals, but instead, I felt guilty. Like I had stolen someone’s spot — someone more qualified and more experienced than me.

I spent the entirety of my 8-week internship battling these feelings of incompetence and fear. I worried that other interns would look at me and think, “How did she scheme her way into this company?” or “How did they let someone like her into the company?” Nevertheless, I shut these thoughts away and focused on creating the best Android app I could with my team. At the end of my internship, I walked away with an app that matched skilled volunteers to non-profit organizations, a return software engineering internship offer, and memories that would last me a lifetime.

However, the impostor syndrome lingered. Now, more than ever before, I felt like I had to prove to my peers at school that my SWE internship offer was legitimate — that I had earned it because of my abilities, not because of my gender. I had to prove to the world that I was truly qualified for the role I had been offered. Moreover, I had to prove to myself that I was enough.

Coming back

I accepted my return offer with the determination to prove others wrong. In May of this year, I started my second internship at Facebook. I would work on the Instagram Growth team as an iOS intern, which was daunting because, for the second summer in a row, I was working on something completely foreign to me.

During the weeks leading up to my internship, I worried that my decision to pursue an iOS role in the company was a mistake. I’d never coded in Objective-C before and was barely more experienced than I was last year. The imposter syndrome crept back in, and it was paralyzing.

Fast forward 12 weeks, and now I’m at the end of my internship here. I’m a different, more confident version of myself. This summer, I spearheaded Instagram’s invites project for empty state surfaces on iOS, developed lasting relationships with the members of my team, and created real impact in the company.

I learned about industry coding standards and workflows, immersed myself in Facebook’s culture. First, a quick summary of my work, and then the lessons I learned.

What did I build?

My work this summer involved adding an invite unit to Direct and Search’s empty state using Instagram’s IGListKit framework. In Direct, I introduced two different user experiences: 1) adding an invite card to the end of the existing collection of activators, and 2) converting all the cards into rows to increase visibility of each of the calls to action:

I also surfaced an invite row in Search’s empty state. When a user searches for an account that doesn’t exist, they are prompted to invite that friend to Instagram:

What did I learn?

1) Be unapologetic

I used to apologize profusely for mistakes in my code or for “wasting” others’ time with my stupid questions. However, apologies fall flat when you keep repeating the same mistakes. Stop apologizing and start acting on feedback instead.

As I was building the invite unit to show to Instagram users in various empty state surfaces across the app, I encountered numerous challenges. My code quality was far from perfect, and my code reviewers often pointed out different areas where I could improve. During my first couple of weeks, I would apologize constantly for these nits, but I soon realized that words are just empty promises if they aren’t backed by actions.

The best way to thank your peers for their time and patience is by showing them your progress. Stop apologizing and, instead, start proving to your peers and managers that their time wasn’t wasted on you.

2) Ask questions — but care enough to listen to the answers

During my first few weeks, the thought of asking another engineer for help terrified me. Fear of judgement crippled me from the start, and the thought of being considered “annoying” held me back even more. I was so obsessed over the idea that everyone would be too busy to entertain me that I found myself writing, deleting, and rewriting messages to my manager when I needed help. I wasted so many hours of my life drafting these messages only to never send them out.

Ultimately, I was so fixated on my own perception of myself that I never even considered how other engineers truly saw me.

It wasn’t until I finally mustered up the courage to ask an iOS engineer on another team for advice that I realized just how eager everyone at Instagram is to help. Never did they give off the impression that I was a nuisance to them or interfering with their everyday work. In fact, they welcomed these opportunities to understand my project and get to know me as a coworker. From that day forward, I stopped drafting and redrafting my messages to my peers and manager. Instead, I just hit send fearlessly because I realized my time was just as valuable as theirs.

However, another crucial thing I learned from asking questions is that asking questions isn’t enough. Listening is the most important part. As humans, we often prepare ourselves to respond while others are still speaking. We don’t take the time to acknowledge others’ words because we’re too adamant that our solutions and thoughts are the only ones that can be right.

That’s an awful way of thinking. Not every conversation is a battle, and not every one of your solutions is the most optimal. The moment I decided to drop that mindset and truly listen to the words of my peers, I learned more than I ever could’ve imagined. So yes, ask your questions, but listen to those answers too.

3) Expand your comfort zone instead of stepping out of it

From the time I was young, I’ve been told to step out of my comfort zone, but it’s never spurred me into action. No one willingly throws themselves into uncomfortable situations, and I’m no different. For the first 8 weeks of my internship, I focused primarily on iOS client-side development. Although I had no previous experience in this area, I slowly but surely grew familiar with the codebase, and gained confidence in my abilities.

However, when my manager Vivian encouraged me to try some server-side work towards the end of my internship, I was intimidated and hesitant to accept. A part of me was fearful of the new learning curve that I’d have to overcome again, this time during my ninth week, when I theoretically should be the most productive. I grew uncertain of my abilities again and was certain that I’d fail and disappoint my team.

As it turns out, working on server code wasn’t all that different from learning the client code. As I did before in my first weeks, I analyzed the existing codebase, contacted other engineers at Instagram for assistance, and threw myself headfirst into the new project. Soon, without realizing it, the server-side code became a part of my comfort zone.

What I’ve discovered is that adopting the mindset of conquering new obstacles instead of abandoning your existing accomplishments has done wonders in helping me tackle new challenges. If we all slowly chip away at the unknown one piece at a time, we can and will prevail. (Even writing this blog post required a lot of courage from me, as I am by no means a professional.)

4) Prove your own worth to yourself first

Perhaps my biggest takeaway from my internship at Instagram is that I am worthy of my position here. From working on the Growth team, I’ve gained a sense for how Instagram products are conceived, built, and delivered. I’ve developed a deep appreciation for all the steps that go into the delivery of a new feature onto our users’ screens. I’ve gained full stack development experience, had measurable impact on the product, and collaborated with a variety of professionals: designers, data scientists, product managers, and more. I’ve taken ownership of a feature and watched as this idea evolved from a design on my computer screen to a fully functioning feature on users’ devices.

None of this would’ve been possible if I didn’t deserve my position here. None of this would’ve been possible if I had only gotten into Facebook because I was a girl. None of this would’ve been possible if I didn’t show my team that I had what it took to succeed as an intern here.

So to all the hopeful interns out there who are confronting the same struggles as I did: stop trying to prove yourself to others when you haven’t even proven yourself to you. Step boldly into your internships and know your own worth because once you silence that nagging voice of doubt inside your head, everything else will fall into place.

And if you can’t quite do that just yet, take a summer to unapologetically expand your comfort zone and surround yourself with people who celebrate your successes with you as I did this summer at Instagram. You’ll be amazed how much you’ll accomplish when you stop doubting your own worth and start believing the people who believe in you.

Thank you!

Ultimately, I’ve grown so much over the course of these past twelve weeks, both technically and personally. Thanks to everyone on the Instagram Growth team for making my summer here an unforgettable one, and special thanks to Vivian Yang and Serena Jiang for their unwavering support. You’ve all helped me believe in myself, and for that I’m eternally grateful.

Amy Huyen was a summer 2019 intern at Instagram. If you want to learn more about this work or are interested in joining one of our engineering teams, please visit our careers page, or follow us on Facebook or on Twitter.

From FBU to SWE — My Journey of Proving Myself to Myself and How Instagram Helped Me Do It was originally published in Instagram Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Kevin Systrom and Mike Krieger founded Instagram in San Francisco in 2010, and now — over seven years and 800 million Instagrammers later — we’re thrilled to be back. The Bay Area has incredible talent, which has helped us create a great and growing team in Menlo Park. Now that Instagram has returned to San Francisco, we’re excited to tap into more talent from across the region.

To start, the San Francisco office will focus on product development from the Creation and Communication team, whose goal is to inspire people to share what matters to them. This is the team behind some of Instagram’s most impactful products, including Stories, Live, Direct, and Profile. Some of the new work we announced at F8, including Camera Effects, Sharing to Stories, and video chat, will be based in San Francisco.

To build these teams, we recruit diverse talent across engineering, product management, design, research, data engineering, and analytics. Our engineering teams are full-stack, and span iOS, Android, and server. Our development teams include graphics experts, and mobile and server infrastructure specialists who build robust systems to power our products at scale, like Live and Direct. The teams in San Francisco are responsible for some of Instagram’s biggest bets, including low-latency video, camera and AR effects, and other rich new user experiences.

Our San Francisco office will continue to expand over the coming months. It’s designed to embody Instagram’s core values, putting community first through art from local and global Instagrammers, inspiring creativity, and simplicity — but with a distinctly San Francisco spin.

Instagram has always operated with a lean team, and we’ve taken a careful approach to growth. Opening a new engineering hub in San Francisco is a big step and something we took very seriously. We know that successful new offices need top, full-stack talent tackling big projects. For example, in our New York City hub, which was established in 2015, we colocated the product and infrastructure teams behind experiences like Home, Discovery, and Shopping, and have since shipped some of Instagram’s most impactful features. Now that we’re in San Francisco, continued collaboration across offices will be more important — and exciting — than ever.

I’m personally incredibly excited to be growing our team in San Francisco. I grew up in the city and have seen it change so much over the years. It’s part of Instagram’s history too, and it’s great to see us take part in the vibrant local community, tap into its creative talent, and be a part of the ever-changing city landscape.

If you want to build the next generation of amazing products at Instagram San Francisco, please join us!

Eddie Ruvinsky is the Engineering Director with the Creation and Communication teams at Instagram San Francisco.

👋 San Francisco! was originally published in Instagram Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Instagram and Facebook, we’ve seen how adopting React Native in product development has allowed our engineers to move and iterate faster on products. Viewing the effects of our changes is now just a single tap of ⌘R away! But working with native code doesn’t provide the same experience. Instead, engineers need to recompile, link, install, and launch the application they’re developing every time they want to see the results of their changes.

Even on a small application, that process drops people out of their workflow. And at our scale, the result is that engineers spend a significant amount of time waiting for their computers to complete work, and often end up context-switching away to other applications, losing even more time.

Late last year we set out to bring the advantages of React Native’s developer experience to engineers working on our native iOS codebase, written in Objective-C, with the goal of allowing engineers to see their changes in seconds. To achieve this, we skip most of the steps necessary to restart the application by loading the new code while it’s still running.

How Native Code Reloading Works

To reload native code, the first step is to turn the modified code into something that we can load into a running application: a dynamic library. This operates on a file-level granularity: we compile only the files that have changed, then link them into a dynamic library. This is very fast — engineers using the tool are primarily iterating on a single file.

An iOS device won’t permit an application to load dynamic libraries from outside of its bundle. Additionally, it won’t allow an application to write to its own bundle. Therefore, all loadable binaries must have been in the application’s bundle when it was installed, and loading our new library is impossible.

Fortunately, neither of these limitations apply to the iOS simulator. Each simulator’s filesystem is simply a directory in host Mac’s filesystem. Additionally, the dlopen function in the iOS simulator doesn't filter by file location, which makes the ability to write directly to the application's bundle irrelevant anyway.

Within the application code, we’ve added a startup function that stars a small, debug-only background task that watches a specific directory. After creating the dynamic library, the tool locates it and moves it into that directory. This triggers a callback in the application, which takes these steps:

• Load the new dynamic library.
• Determine which Objective-C classes have been added.
• Find the original versions of these classes. Objective-C classes are themselves objects, and they’re easy to look up by name.
• Use the Objective-C runtime APIs to replace the implementations of methods on the original versions of the classes with the new implementations. This applies to all methods of the class, as we don’t have any way to determine which have changed and which haven’t.
• Write a response back to the same directory for the tool to read.

Our new code is now running in the application, and will be called whenever one of these methods is next invoked. Of course, this is only available while in development with the iOS simulator, not on the device or in the production app.

Building a Developer Experience

Reinstalling and restarting an application to see new changes isn’t fast, but it’s simple and reliable. Click a button or press a keyboard shortcut in Xcode, and eventually, the results of your code will appear on-screen. We match this in our tooling: we place a button in Xcode’s toolbar, and add a keyboard shortcut to trigger the loading of new code.

We have an another advantage over full restarts: we can keep all of the current application state. A restart will reset the application back to its entry point — for our apps, that’s generally a feed. Engineers working on other products will have to click their way back to their products’ views before they can see if their changes worked correctly.

However, after new code has been loaded, nothing happens: all of the previous state that has been built up is unaffected by the engineer’s changes. Fortunately, ComponentKit, a declarative iOS UI framework used at Facebook, has a useful method for performing an instant relayout: +[CKComponentDebugController reflowComponents]. Views built with ComponentKit can invoke it after new code is loaded, causing their new layout code to be evaluated and the on-screen views to update immediately. The result is that when working on UI code written with ComponentKit, engineers can make changes to their view layouts, and see them within seconds. Previously, engineers would need to batch their updates and think about when they wanted a full restart. They can now work interactively and iteratively.

Conclusion

Our tool has become popular with our teams here, especially among engineers working with ComponentKit. In larger apps, it has allowed engineers to see the results of their changes up to twenty times faster. The combination of fast performance and tight integration with existing developer tools and frameworks has made native code reloading an important part of many iOS engineers’ workflows here at Instagram and Facebook.

Nate Stedman is an iOS engineer at Instagram New York.

Instant Feedback in iOS Engineering Workflows was originally published in Instagram Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Making Type Mode function seamlessly on both iOS and Android had its own set of challenges, including dynamically resizing text and custom background fills. In this post we’ll take a look at how we approached this work on both iOS and Android platforms.

Dynamically Resizing Text Input

With Type Mode, we wanted to create a text input experience that let people emphasize certain words or phrases. One way to do that was to build fully justified text styles that dynamically resize each line to fill the available width (used in Modern, Neon, and Strong).

iOS

The main challenge on iOS was to render dynamically resizing text in a native UITextView, which would let people enter text in a quick and familiar way.

RESIZING PRE-TEXT STORAGE COMMIT

While you input text on a line, the size of text on that line should scale down until it hits some minimum font size.

To accomplish this, we use a combination of UITextView.typingAttributes, NSAttributedString, and NSLayoutManager.

First, we need to calculate what font size our text will be rendered as. We can grab the range of the line we are currently typing on using [NSLayoutManager enumerateLineFragmentsForGlyphRange:usingBlock:]. From that range we can create a sizing string to calculate a minimum font size.

CGFloat pointSize = 24.0; // arbitrary
NSAttributedString *attributedString = [[NSAttributedString alloc] initWithString:string attributes:@{NSFontAttributeName:[UIFont fontWithName:fontName size:pointSize]}];
CGFloat textWidth = CGRectGetWidth([attributedString boundingRectWithSize:CGSizeMake(CGFLOAT_MAX, CGFLOAT_MAX) options:NULL context:nil]);
CGFloat scaleFactor = (textViewContainerWidth / textWidth);
CGFloat preferredFontSize = (pointSize * scaleFactor);
return CLAMP_MIN_MAX(preferredFontSize, minimumFontSize, maximumFontSize) // Clamps value between min and max

In order to actually draw the text at the correct size, we need to use our new font size in our UITextView's typingAttributes. UITextView.typingAttributes are attributes that apply to new text being typed by the user. A good place for this is [id<UITextViewDelegate> textView:shouldChangeTextInRange:replacementText:].

- (BOOL)textView:(UITextView *)textView shouldChangeTextInRange:(NSRange)range replacementText:(NSString *)text {
    NSMutableDictionary *typingAttributes = [textView.typingAttributes mutableCopy];
    typingAttributes[NSFontAttributeName] = [UIFont fontWithDescriptor:fontDescriptor size:calculatedFontSize];
    textView.typingAttributes = typingAttributes;
    return YES;
}

This means that as the user types, the font size will shrink until we hit some specified minimum. When this happens, the UITextView will wrap our text as it normally would.

FINALIZE TEXT POST STORAGE COMMIT

After our text has been committed to text storage, we may need to clean up some sizing attributes. Our text could have wrapped, or the user could have “emphasized” text by manually adding a line break to write larger text on a separate line.

A good place to put this logic is in [id<UITextViewDelegate> textViewDidChange:]. This happens after the text has been committed to text storage and initially laid out by the text engine.

To get a list of character ranges for each line, we can use NSLayoutManager:

NSMutableArray<NSValue *> *lineRanges = [NSMutableArray array];
[textView.layoutManager enumerateLineFragmentsForGlyphRange:NSMakeRange(0, layoutManager.numberOfGlyphs) usingBlock:^(CGRect rect, CGRect usedRect, NSTextContainer * _Nonnull textContainer, NSRange glyphRange, BOOL * _Nonnull stop) {
    NSRange characterRange = [layoutManager characterRangeForGlyphRange:glyphRange actualGlyphRange:NULL];
    [lineRanges addObject:[NSValue valueWithRange:characterRange]];
}];

We then need to manipulate the NSTextStorage by setting attributes on ranges that have the correct font size for each row.

There are three stages to editing NSTextStorage, which is itself a subclass of NSMutableAttributedString.

1. Call [textStorage beginEditing] to indicate we are making one or more changes to the text storage.
2. Send some editing messages to NSTextStorage. In our case, the NSFontAttributeName attribute should be set with the correct font size for that row. We can use a similar method for calculating the font size as we did earlier.

for (NSValue *lineRangeValue in lineRanges) {
    NSRange lineRange = lineRangeValue.rangeValue;
    const CGFloat fontSize = ... // Same font size calculation from earlier
    [textStorage setAttributes:@{NSFontAttributeName : [UIFont fontWithDescriptor:fontDescriptor size:fontSize]} range:lineRange];
}

3. Call [textStorage endEditing] to indicate we are done editing text storage. This invokes the [NSTextStorage processEditing] method which will fix attributes on the ranges we changed. This also invokes the correct NSTextStorageDelegate methods.

TextKit is a powerful and modern API that is tightly integrated with UIKit. Many text experiences can be designed with it, and new text APIs are being released in most new iOS versions. With TextKit, you can do anything from creating custom text containers to modifying the actual generation of glyphs. And since it is built on top of CoreText and integrated with APIs like UITextView, text input and editing continues to feel like a native iOS experience.

Android

Android doesn’t support full width justification out of the box, but the framework APIs give us all the tools we need to implement it ourselves.

The first step is to lay the text out at the minimum text size. We’ll scale things up later, but this will tell us how many lines we have and where the line breaks are:

TextPaint textPaint = new TextPaint();
textPaint.setTextSize(SIZE_MIN);
Layout layout =
    new StaticLayout(
        text,
        textPaint,
        availableWidth,
        Layout.Alignment.ALIGN_CENTER,
        1 /* spacingMult */,
        0 /* spacingAdd */,
        true /*includePad */);
int lineCount = layout.getLineCount();

Next we need to go through the layout and resize each line individually. There’s no direct way to find the perfect text size for a particular line, but we can easily approximate it with a binary search for the largest text size that doesn’t force a line break:

int lowSize = SIZE_MIN;
int highSize = SIZE_MAX;
int currentSize = lowSize + (int) Math.floor((highSize - lowSize) / 2f);
while (low < current) {
  if (hasLineBreak(text, currentSize)) {
    highSize = currentSize;
  } else {
    lowSize = currentSize;
  }
  currentSize = lowSize + (int) Math.floor((highSize - lowSize) / 2f);
}

Once we’ve found the right size for each line, we apply it as a span. Spans let us use different text sizes for each line instead of a single text size for the entire string:

text.setSpan(
    new AbsoluteSizeSpan(textSize),
    layout.getLineStart(lineNumber),
    layout.getLineEnd(lineNumber),
    Spanned.SPAN_INCLUSIVE_EXCLUSIVE);

Now each line of text fills the available width! We can repeat this process each time the text changes to get the dynamic resizing behavior we’re looking for.

Custom Backgrounds

We also wanted Type Mode to let people emphasize words and phrases with text backgrounds (used in Typewriter and Strong).

iOS

Another way we can leverage NSLayoutManager is for drawing custom background fills. NSAttributedString does have a NSBackgroundColorAttributeName, but it is not customizable nor extensible.

For example, if we used NSBackgroundColorAttributeName, the background of the entire text view would be filled in. We could not exclude whitespace, have spaces between the lines, or draw the fill with a corner radius. Thankfully, NSLayoutManager lets us override drawing the background fill. We need to create a NSLayoutManager subclass and override drawBackgroundForGlyphRange:atPoint:

@interface IGSomeCustomLayoutManager : NSLayoutManager
@end 

@implementation IGSomeCustomLayoutManager

- (void)drawBackgroundForGlyphRange:(NSRange)glyphsToShow atPoint:(CGPoint)origin {
    // Draw custom background fill

    [super drawBackgroundForGlyphRange:glyphsToShow atPoint:origin];
}
    
}];
@end

With our drawBackgroundForGlyphRange:atPoint method, we can once again leverage [NSLayoutManager enumerateLineFragmentsForGlyphRange:usingBlock] to grab the glyph ranges of each line fragment. We can then use [NSLayoutManager boundingRectForGlyphRange:inTextContainer] to get the bounding rectangle of each line.

- (void)drawBackgroundForGlyphRange:(NSRange)glyphsToShow atPoint:(CGPoint)origin {
  [self enumerateLineFragmentsForGlyphRange:NSMakeRange(0, self.numberOfGlyphs) usingBlock:^(CGRect rect, CGRect usedRect, NSTextContainer * _Nonnull textContainer, NSRange glyphRange, BOOL * _Nonnull stop) {
       CGRect lineBoundingRect = [self boundingRectForGlyphRange:glyphRange inTextContainer:textContainer];
       CGRect adjustedLineRect = CGRectOffset(lineBoundingRect, origin.x + kSomePadding, origin.y + kSomePadding);
       UIBezierPath *fillColorPath = [UIBezierPath bezierPathWithRoundedRect:adjustedLineRect cornerRadius:kSomeCornerRadius];
       [[UIColor redColor] setFill];
       [fillColorPath fill];
  }];
}

This allows us to draw a background fill around any arbitrary text with our own specified shapes and padding. NSLayoutManager can also be utilized to draw other text attributes like strikethroughs and underlines.

Android

At first glance, it feels like this should be simple to implement on Android. We can add a span that modifies the text background color and be done:

new CharacterStyle() {
  @Override
  public void updateDrawState(TextPaint textPaint) {
    textPaint.bgColor = color;
  }
}

That’s a good first attempt (and was the first thing we built), but it comes with a few limitations:

1. The background tightly wraps the text and there’s no way to adjust the padding.
2. The background is rectangular and there’s no way to adjust the corner radius.

To address those issues, we tried using LineBackgroundSpan. We were already using it to render the rounded bubble background on Classic text, so it seemed like a natural fit for the new text styles as well. Unfortunately our new use case uncovered a subtle bug in the framework Layout class. If your text has multiple LineBackgroundSpan instances on different lines, then Layout will not iterate through them properly and some of them may never be rendered.

Thankfully we can sidestep the framework bug by applying a single LineBackgroundSpan to the entire string and then delegating to individual background spans ourselves:

class BackgroundCoordinator implements LineBackgroundSpan {
  @Override
  public void drawBackground(
      Canvas canvas,
      Paint paint,
      int left,
      int right,
      int top,
      int baseline,
      int bottom,
      CharSequence text,
      int start,
      int end,
      int currentLine) {
    Spanned spanned = (Spanned) text;
    for (BackgroundSpan span : spanned.getSpans(start, end, BackgroundSpan.class)) {
      span.draw(canvas, spanned);
    }
  }
}

class BackgroundSpan {
  public void draw(Canvas canvas, Spanned spanned) {
    // Custom background rendering...
  }
}

Conclusion

Instagram has a very strong prototyping culture, and the design team’s Type Mode prototypes let us get a real feel for the user experience with each iteration along the way. For example, with the Neon style, we needed a way to take a single color from our palette and then generate an interior color and a glow color for the text. A designer on the project played around with some approaches in his prototype, and when he found one he liked we essentially just copied his logic on Android and iOS. This level of collaboration with the design team was a special part of this launch and made the development process really efficient.

If you’re interested in working with us on Stories, check out our careers page for roles in Menlo Park, New York, and San Francisco.

Christopher Wendel and Patrick Theisen are iOS and Android engineers at Instagram, respectively.

Building Type Mode for Stories on iOS and Android was originally published in Instagram Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Motivation

At Instagram, we use Apache Cassandra heavily as a general key value storage service. The majority of Instagram’s Cassandra requests are online, so in order to provide a reliable and responsive user experience for hundreds of millions of Instagram users, we have very tight SLA on the metrics.

Instagram maintains a 5–9s reliability SLA, which means at any given time, the request failure rate should be less than 0.001%. For performance, we actively monitor the throughput and latency of different Cassandra clusters, especially the P99 read latency.

Here’s a graph that shows the client-side latency of one production Cassandra cluster. The blue line is the average read latency (5ms) and the orange line is the P99 read latency (in the range of 25ms to 60ms and changing a lot based on client traffic).

After investigation, we found the JVM garbage collector (GC) contributed a lot to the latency spikes. We defined a metric called GC stall percentage to measure the percentage of time a Cassandra server was doing stop-the-world GC (Young Gen GC) and could not serve client requests. Here’s another graph that shows the GC stall percentage on our production Cassandra servers. It was 1.25% during the lowest traffic time windows, and could be as high as 2.5% during peak hours.

The graph shows that a Cassandra server instance could spend 2.5% of runtime on garbage collections instead of serving client requests. The GC overhead obviously had a big impact on our P99 latency, so if we could lower the GC stall percentage, we would be able to reduce our P99 latency significantly.

Solution

Apache Cassandra is a distributed database with it’s own LSM tree-based storage engine written in Java. We found that the components in the storage engine, like memtable, compaction, read/write path, etc., created a lot of objects in the Java heap and generated a lot of overhead to JVM. To reduce the GC impact from the storage engine, we considered different approaches and ultimately decided to develop a C++ storage engine to replace existing ones.

We did not want to build a new storage engine from scratch, so we decided to build the new storage engine on top of RocksDB.

RocksDB is an open source, high-performance embedded database for key-value data. It’s written in C++, and provides official API language bindings for C++, C, and Java. RocksDB is optimized for performance, especially on fast storage like SSD. It’s widely used in the industry as the storage engine for MySQL, mongoDB, and other popular databases.

Challenges

We overcame three main challenges when implementing the new storage engine on RocksDB.

The first challenge was that Cassandra does not have a pluggable storage engine architecture yet, which means the existing storage engine is coupled together with other components in the database. To find a balance between massive refactoring and quick iterations, we defined a new storage engine API, including the most common read/write and streaming interfaces. This way we could implement the new storage engine behind the API and inject it into the related code paths inside Cassandra.

Secondly, Cassandra supports rich data types and table schema, while RocksDB provides purely key-value interfaces. We carefully defined the encoding/decoding algorithms to support Cassandra’s data model within RocksDB’s data structure and supported same-query semantics as original Cassandra.

The third challenge was about streaming. Streaming is an important component for a distributed database like Cassandra. Whenever we join or remove a node from a Cassandra cluster, Cassandra needs to stream data among different nodes to balance the load across the cluster. The existing streaming implementation was based on the details in the current storage engine. Accordingly, we had to decouple them from each other, make an abstraction layer, and re-implement the streaming using RocksDB APIs. For high streaming throughput, we now stream data into temp sst files first, and then use the RocksDB ingest file API to bulk load them into the RocksDB instance at once.

Performance metrics

After about a year of development and testing, we have finished a first version of the implementation and successfully rolled it into several production Cassandra clusters in Instagram. In one of our production clusters, the P99 read latency dropped from 60ms to 20ms. We also observed that the GC stalls on that cluster dropped from 2.5% to 0.3%, which was a 10X reduction!

We also wanted to verify whether Rocksandra would perform well in a public cloud environment. We setup a Cassandra cluster in an AWS environment using three i3.8 xlarge EC2 instances, each with 32 cores CPU, 244GB memory, and raid0 with 4 nvme flash disks.

We used NDBench for the benchmark, and the default table schema in the framework:

TABLE emp (
emp_uname text PRIMARY KEY,
emp_dept text,
emp_first text,
emp_last text
)

We pre-loaded 250M 6KB rows into the database (each server stores about 500GB data on disk). We configured 128 readers and 128 writers in NDBench.

We tested different workloads and measured the avg/P99/P999 read/write latencies. As you can see, Rocksandra provided much lower and consistent tail read/write latency.

We also tested a read-only workload and observed that, at similar P99 read latency (2ms), Rocksandra could provide 10X higher read throughput (300K/s for Rocksandra vs. 30K/s for C* 3.0).

Future work

We have open sourced our Rocksandra code base and benchmark framework, which you can download from Github to try out in your own environment! Please let us know how it performs.

As our next step, we are actively working on the development of more C* features support, like secondary indexes, repair, etc. We are also working on a C* pluggable storage engine architecture to contribute our work back to the Apache Cassandra community.

If you are in the Bay Area and are interested in learning more about our Cassandra developments, join us at our next meetup event here.

Dikang Gu is an infrastructure engineer at Instagram.

Open-sourcing a 10x reduction in Apache Cassandra tail latency was originally published in Instagram Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

From our load test, we could see that memory usage was becoming our bottleneck. Enabling GC could alleviate this problem and slow down the memory growth, but undesired Copy-on-write (COW) would still increase the overall memory footprint. So we decided to see if we could make Python GC work without COW, and hence, the memory overhead.

First try: redesign the GC head data structure

If you read our last GC post carefully, you’ll notice the culprit of the COW was ahead of each python object:

/* GC information is stored BEFORE the object structure. */
typedef union _gc_head 
{
    struct {
        union _gc_head *gc_next;
        union _gc_head *gc_prev;
        Py_ssize_t gc_refs;
    } gc;
    long double dummy; /* force worst-case alignment */
} PyGC_Head;

The theory was that every time we did a collection, it would update the gc_refs with ob_refcnt for all tracked objects — but unfortunately this write operation caused memory pages to be COW-ed. A next obvious solution was to move all the head to another chunk of memory and store densely.

We implemented a version where the pointer in the gc_head struct didn’t change during collection:

typedef union _gc_head_ptr
{
    struct {
        union _gc_head *head;
    } gc_ptr;
    double dummy; /* force worst-case alignment */
} PyGC_Head_Ptr;

Did it work? We used the following script to allocate the memory and fork a child process to test it:

lists = []
strs = []

for i in range(16000):
    lists.append([])
    for j in range(40):
        strs.append(' ' * 8)

With the old gc_head struct, the child process’s RSS memory usage increased by ~60MB. Under the new data structure with the additional pointer, it only increased by ~0.9 MB. So it worked!

However, you may have noticed the additional pointer in the proposed data structure introduced memory overhead (16 bytes — two pointers). It seems like a small number, but if you consider it applied to every collectable Python object (and we usually have millions of objects in one process, with ~70 processes per host), it could be a fairly big memory overhead on each server.

16 bytes* 1,000,000 * 70 = ~1 GB

Second try: hiding shared objects from GC

Even though the new gc_head data structure showed promising gains on memory size, its overhead was not ideal. We wanted to find a solution that could enable the GC without noticeable performance impacts. Since our problem is really only on the shared objects that are created in the master process before the child processes are forked, we tried letting Python GC treat those shared objects differently. In other words, if we could hide the shared objects from the GC mechanism so they wouldn’t be examined in the GC collection cycle, our problem would be solved.

For that purpose, we added a simple API as gc.freeze() into the Python GC module to remove the objects from the Python GC generation list that‘s maintained by Python internal for tracking objects for collection. We have upstreamed this change to Python and the new API will be available in the Python3.7 release (https://github.com/python/cpython/pull/3705).

static PyObject *
gc_freeze_impl(PyObject *module)
{
    for (int i = 0; i < NUM_GENERATIONS; ++i) {
        gc_list_merge(GEN_HEAD(i), &_PyRuntime.gc.permanent_generation.head);
        _PyRuntime.gc.generations[i].count = 0;
    }
    Py_RETURN_NONE;
}

Success!

We deployed this change into our production and this time it worked as expected: COW no longer happened and shared memory stayed the same, while average memory growth per request dropped ~50%. The plot below shows how enabling GC helped the memory growth by stopping the linear growth and making each process live longer.

Credits

Thanks to Jiahao Li, Matt Page, David Callahan, Carl S. Shapiro, and Chenyang Wu for their discussions and contributions to the COW-friendly Python garbage collection.

Zekun Li is an infrastructure engineer at Instagram.

Copy-on-write friendly Python garbage collection was originally published in Instagram Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Motivation

At Instagram we have hundreds of engineers working on well over a million lines of Python 3. Every day we have new engineers joining the team from other projects and other languages who need to ramp up quickly and get productive in our codebase. And we’re constantly shipping new code into production, every few minutes, all day long, every day. So we’re keen to make our code easier for new developers to read and understand, as well as more amenable to static analysis that shrinks the domain of possible bugs. Type annotations and static type checking fit that bill.

Writing new code with type annotations is easy enough; most of our engineers are eager to do that. But the returns on static type checking are low until we reach a critical mass of type-annotated code, especially our core frameworks and libraries. In other words: we have a lot of existing code that needs type annotations added.

Our first forays into manually adding type annotations were discouraging. It can take hours to annotate a single module, sometimes painstakingly tracing through multiple layers of function calls and objects to understand the possible types at some call site. (This is, of course, the same pain that anyone trying to maintain that function might experience; that’s why we want to add type annotations!)

So we built MonkeyType. Instead of guessing or spelunking for the right types, let your test suite or (better!) your production system tell you what the real types are.

Usage

Sounds great! I’ve run pip install monkeytype. What's next?

Before MonkeyType can tell us anything useful, we need to let it trace some function calls. The simplest way to do this is with monkeytype run, which runs any Python script under MonkeyType tracing. For instance, you can easily run your test suite under MonkeyType:

$ monkeytype run runtests.py

(or monkeytype run `which pytest`, or whatever your preferred flavor.)

While your tests are running, MonkeyType inspects the argument types and return/yield type of every function call, recording them in a database. (By default it keeps them in a local SQLite database, but like just about everything MonkeyType does, this is configurable.)

Of course, your test suite may not provide the best type information — sometimes tests use fakes instead of the real types, and we’ve found plenty of cases where type checking revealed that our tests were accidentally passing in different types from production. So if you don’t want to annotate based on your test suite, you can record call traces from production runtime. For this use case, MonkeyType provides a context manager API:

from monkeytype import trace

with trace():
    # ...

If you need even more flexibility, you can create your own CallTracer, install it with sys.setprofile(), and remove it when you're ready.

Once you've got some call traces recorded, you can generate a stub file for any module:

$ monkeytype stub some.module

If the stub looks reasonable and you want to apply the annotations directly to your code, MonkeyType will do that too:

$ monkeytype apply some.module

Review the now-annotated code in some/module.py, correct the annotations if needed, and commit it!

Because of the backing-store design, you don't have to record traces and annotate all at one go. You can collect traces into the database over a long period of time and gradually annotate more and more modules from the collected data as you're ready to do so.

With configurable type rewriters, you can easily tweak MonkeyType’s generated type annotations for your preferred type annotation style or specific edge cases in your codebase.

For lots more details on customization options, refer to the documentation.

Open Source

Check out the code, give it a spin, and let us know what you think! We’re looking forward to your bug reports and suggested improvements. If you’d like to contribute, we have a list of suggested starter tasks in the issue tracker.

MonkeyType does require Python 3.6+, and generates only Python 3 style type annotations (no type comments). If you aren’t quite there yet, you may want to start with our PyCon 2017 keynote on how we migrated those million lines of code to Python 3 last year.

How we use MonkeyType at Instagram

We choose a small random sample of production web requests and turn on MonkeyType tracing via a Django middleware. Traces are stored and retrieved from SCUBA, Facebook’s data analysis platform, using a custom CallTraceStore. We run tracing constantly, so we are always adding new type traces from production. Since production code changes frequently, this keeps our traces up to date.

Whenever an engineer wants to add type annotations to a module, they just run monkeytype stub and then monkeytype apply, fix any type errors revealed by the new annotations, and submit a diff!

Results

With MonkeyType’s help, we’ve already annotated over a third of the functions in our codebase, and we’re already seeing type-checking catch many bugs that would have otherwise likely shipped to production. Race you to 100%!

Carl Meyer is an engineer on Instagram’s infrastructure team.

Let your code type-hint itself: introducing open source MonkeyType was originally published in Instagram Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Getting started

To determine what we needed to optimize and which techniques we should use, we had to figure out:

• What the Instagram workload looked like from the interpreter’s perspective, and
• Where the interpreter spent its time when it was running our code.

We started by collecting various bits of data from CPython as it executed Instagram server code. Unsurprisingly, from the interpreter’s perspective, the Instagram workload looks like an object-oriented, web server workload:

• 90% of instructions by the interpreter dealt with operand stack manipulation, control flow, and attribute access.
• Our workload wasn’t loopy — 94% of loops terminated after four or fewer iterations.
• Function call overhead was quite high and accounted for roughly 30% of the time spent by the interpreter.
• Attribute access was the next most resource-intensive category of opcodes, and accounted for another 28% of time spent in the interpreter. Attribute loads were not particularly polymorphic; 85% of loads occurred at monomorphic sites.

Based on the data above, we decided to work to reduce the computational cost of function calls and attribute access.

Data Collection Methodology

We collected all of our data by instrumenting an interpreter running in our lab environment, InstaLab. InstaLab replays production traffic to web servers that are running in an isolated environment using a request mix taken from the top five views (by CPU instructions). It provides us with a way to collect representative performance data without fear of negatively impacting user experience. For each different data set, we instrumented CPython, built a new interpreter, and collected the relevant data from InstaLab.

Bytecode frequency distribution

From the perspective of the interpreter, Instagram is just a sequence of bytecode instructions to be executed. As a first step in understanding what the interpreter was doing, we instrumented CPython to collect bytecode execution frequencies. The top ten instructions by frequency are shown in the graph below:

Immediately, we could see that LOAD_FAST dominated. In fact, LOAD_FAST, STORE_FAST and LOAD_CONST accounted for roughly 40% of all opcodes executed. This was expected — CPython is a stack machine and each of these instructions moves values to/from the operand stack. Although these instructions are cheap, they are executed quite frequently, and generate memory traffic. This suggested techniques to eliminate loads and stores (e.g. switching to a register based bytecode) would be an effective optimization.

When we excluded the opcodes above, we could get a better sense of the “real work” done by the interpreter when it executed Instagram code. The following graph shows the distribution of execution frequency for the top twenty opcodes (of 120 total).

These instructions accounted for 90% of the remaining distribution. We grouped them broadly into categories:

• Attribute access — LOAD_ATTR, STORE_ATTR, LOAD_GLOBAL
• Control flow — CALL_FUNCTION, RETURN_VALUE, POP_JUMP_IF_FALSE, COMPARE_OP, FOR_ITER, JUMP_ABSOLUTE, POP_JUMP_IF_TRUE, POP_BLOCK, JUMP_FORWARD, YIELD_VALUE, CALL_FUNCTION_KW, SETUP_EXCEPT
• Container Manipulation — BINARY_SUBSCR, BUILD_TUPLE, UNPACK_SEQUENCE
• Operand Stack Manipulation — POP_TOP, LOAD_DEREF

This conformed to what we expected to see from an object-oriented, web server workload (as opposed to a numeric workload).

Time spent executing bytecode

Looking at execution frequencies only told part of the story. In addition to knowing what the interpreter was doing, we also wanted to know where the interpreter spent most of its time. To that end, we measured the number of CPU instructions that were retired while executing each opcode using the perf_event APIs. We bracketed each opcode body in the interpreter dispatch loop with a read of a hardware instruction counter, then subtracted the two values to compute the “time” spent executing the opcode. For cases where calls were made back into the interpreter, or out to C functions (e.g. the CALL_FUNCTION family of opcodes), we deducted the amount of “time” spent in the callee from the cost attributed to the opcode.

The graph below shows the top 10 opcodes by percentage of cumulative CPU instructions retired.

This data painted a slightly different picture, as function calls and attribute loading jumped to the head of the distribution. In terms of “time spent,” the most resource-intensive categories of opcodes were:

• Control flow — CALL_FUNCTION, CALL_FUNCTION_KW, COMPARE_OP
• Attribute loading — LOAD_ATTR, STORE_ATTR, LOAD_GLOBAL
• Operand stack manipulation — LOAD_FAST, STORE_FAST, LOAD_CONST

Based on the data above, as a first step we decided to spend time optimizing attribute access and reducing function call overhead.

Accelerating attribute access

One time-honored technique to accelerate attribute access and method dispatch in dynamic language runtimes is the polymorphic inline cache. Inline caches, in combination with dense attribute storage (using something like hidden classes), can dramatically speed up attribute lookup. However, their efficacy depends on the degree of polymorphism seen at the call sites.

We instrumented CPython to record the types seen for each execution of LOAD_ATTR in order to gauge the potential of inline caching. The graph below shows the distribution of LOAD_ATTRs by degree of polymorphism.

Based on the data above, inline caching looked like it would be an effective technique to accelerate attribute access. Roughly 85% of LOAD_ATTRs occured at monomorphic sites; an inline cache of size four should have been able to handle roughly 96% of attribute loads.

Workload loopyness

The final question we wanted to answer was: “how loopy is our code?” Answering this question helped us determine how effective optimizations targeted at reducing loop overhead would be at accelerating our workload.

We instrumented CPython to record the number of iterations performed by each instance of a loop. The graph below shows the distribution of loop instances grouped by the number of iterations performed.

As we can see, our workload was not loopy. In fact, 94% of loops terminated after four or fewer iterations.

Conclusion

As a result of this exercise we identified two major sources of inefficiency for our workload in CPython: function call overhead and attribute access requirements. The data we’ve collected suggests that well-known techniques should be effective at mitigating the inefficiency. As a next step, we’re using this information to guide our efforts to optimize CPython.

Matt Page is a software engineer on Instagram’s efficiency and reliability team.

Profiling CPython at Instagram was originally published in Instagram Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

• increase in application size downloaded from an app store
• increase in cold start time, and more generally, increased time to interaction with different surfaces inside the app
• increase in disk space usage
• decrease in developer velocity from increased complexity of the app or build time

To tackle these issues, we recently enacted an Instagram-wide code modularization effort to create strict isolation between features. We do this on both Android and iOS, but this post will primarily focus on Android. Once a feature is modularized, we can take the feature out of the primary executable file and compile it to the dedicated file and lazily load it during app runtime on-demand. Lazy loading can bring a combination of benefits like improved cold start time (especially on Dalvik) and overall runtime, decreased disk footprint, and better developer velocity if module hotswapping is implemented. This post will walk through how we approach modularization and lazy loading at Instagram, and how you can implement it in your own app with our new open-sourced framework.

App modularization

What is modularization

Modularization is the process of separating and creating clear boundaries between logical components of code. For Instagram, we’ve been focused on modularizing feature code since it represents a large part of the codebase, and each feature has clear boundaries. Reducing cross-feature dependencies with modularization helps us solve the challenges of a growing app in a couple of ways. One benefit is that it helps us protect our app’s start time from growing as the amount of code in the app increases. Before modularization, a chain of code references from one feature to another might have risked loading in all of that code during app start time. But by breaking apart these references with modularization, we can wait to load code when it’s actually needed using methods like Lazy Loading (see below).

In addition, modularization has other great benefits for overall codebase sanity, build times, developer velocity, Instant Apps, etc.

How to modularize

The goal of modularization is to have a clearly defined interface that separates the implementation of the feature from outside code. Start by auditing all the incoming dependencies to the feature. For each dependency, figure out if it should be removed or kept. Since each case is different, there is no simple way of making this evaluation — but here are some considerations to make:

• Is the dependency a result of misplaced code? Sometimes a dependency exists because there’s a shared library method or misplaced method in the feature that should actually live outside of the feature. If so, move the referenced code out of the feature and into the appropriate place.
• Is there a way to rewrite the dependency to be more generic rather than have it reference feature-specific code?
• Does the dependency exist because we’re accessing some feature-specific logic earlier than necessary? If so, try to defer this logic until later.
• Is the surrounding logic overloaded? If so, simplify the logic into the feature-specific part to make it simpler to reason about the necessity of the dependency.

If the dependency is still necessary after making these considerations, then it belongs as part of the feature’s interface. If we’ve done the modularization well, the feature’s interface should contain just a few critical methods. Some examples are:

• Methods that allow navigating to the feature
• Lifecycle methods that listen for app-wide lifecycle events
• Methods that expose core functionality in the feature.

Module lazy loading

What is lazy loading

While modularization will decouple a feature from the outside world and create a clean interface to the feature, lazy loading will take this further and compile the feature out of the main executable (dex) file to a separate dex file. Each lazily loaded feature will live in a separate file, which will bring benefits like:

• allowing the feature to load in memory only when really needed instead of on every cold start. It offloads code from the main executable file, which remains smaller and guarantees better cold start time. On Dalvik it offloads methods off the main dex file, decreasing performance penalty of multi dex.
• clustering feature code together in memory, as it lives in one file and provides most optimal execution in terms of memory access
• using less disk space if some modules remain unused because feature code will never be uncompressed
• possibility of having an adaptable app with different sets of features shipped to different users (e.g. non-business users don’t need business features), thus decreasing initial app size

And for developer velocity, we added support for hotswapping of lazy loaded modules, which means developers will be able to keep coding and seeing changes without restarting the app.

When we trigger lazy loading

Generally speaking we load a module when we expect it to be used in the near future. Once the module is loaded it will remain in memory until the next cold start. Module loading may incur a small latency (depending on the size of the module), so different tactics may be applied:

• Loading a module in the background when the user is one click away from the module. This may mean that the module remains unused if the user doesn’t decide to click into that feature or navigates back. But if there’s high probability of clicking into that module, then it’s a good solution.
• Loading a module once the user navigates into the module. If loading latency is small (below 50ms) for the majority of cases (e.g. p99), then we could simply block on loading, and once it’s done navigate into the feature. Otherwise a simple spinner or a progress bar can be displayed so that the app does not appear as frozen.
• Some modules are by nature asynchronous, which eases lazy loading since it will be the part of the asynchronous loading. An example is a video player that runs in a secondary process. Instagram initially shows a screenshot of the video while it loads in the background (often times fetched from the network). Lazy loading would happen in that secondary process and be completely transparent to users.

We open sourced our framework for lazy loading at ig-lazy-module-loader so that other developers can use it in their apps, or simply learn from our implementation. It can be beneficial if you’re hitting a performance penalty of multi-dex on Dalvik or have many features which may remain unused by some users and you want to save disk space.

We recently moved our mobile infrastructure engineering teams (iOS and Android) to New York City. If this blog post got you excited about what we’re doing, we’re hiring — check out our careers page.

Mona Huang and Julian Krzeminski work on Android at Instagram.

App modularization and module lazy loading at Instagram and beyond was originally published in Instagram Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.