How Ladies Lead Knowledge Engineering at Slack

The Knowledge Engineering workforce is answerable for Slack’s information lake, analytics dashboards, and different information providers. The workforce’s mission is to empower customers to leverage information to make choices shortly, precisely, and simply. Slack’s information lake grew in measurement from sub-petabyte to over 100 petabytes lately and it now spans tens of millions of tables. Because the complexity of managing this information grew, so did a various workforce of Slack engineers devoted to supporting the ecosystem. 

We’ve got sturdy feminine illustration amongst engineers in Knowledge at Slack. Our Knowledge Engineering tradition celebrates variety in views and experiences. As information complexity intensifies, having a mosaic of artistic drawback solvers from completely different backgrounds is the important thing to navigating intricate challenges with agility and perception.

Let’s dive into the private tales of girls who’re redefining Slack’s information panorama.

  • Hearken to Jessica, a current rent, as she navigates the complexities of Pinot and Tableau know-how.
  • Observe Nilanjana, Ramya, Shrushti, and Nathalie, seasoned engineers, as they lead large-scale engineering initiatives utilizing applied sciences like Spark, Merlin, DataHub, and Secor.
  • Lastly, hear from Suzanna, Lakshmi, and Beate, feminine engineering leaders, as they make clear the panorama of development and alternatives for girls in information engineering. 

Hello, I’m Jessica Stewart, and I’m a Senior Software program Engineer on the Knowledge Orchestration workforce. I joined in Could 2023.

My workforce oversees the inner implementations of information workflows utilizing Apache Airflow and Apache Pinot. Working inside our major cluster, which shops terabytes of information, we preserve a system that boasts sub-second question latency and an almost 99.95% question success charge service-level settlement (SLA). This datastore serves us internally by supporting instruments for workers and externally because the spine for Slack’s user-facing analytics dashboards.

A present technical problem we face revolves round migrating from a digital machine setup to a cloud-native Kubernetes infrastructure. Digital machines imply greater infrastructure prices and upkeep overhead, so we had been keen to extend effectivity with a containerized set-up. We’ve got an inside Slack Kubernetes platform that doesn’t totally assist Helm charts or Kubernetes providers, so we needed to make some customizations to the open-source answer. To be able to preserve efficiency necessities, we configured a buyer networking set-up that integrates Pinot with our inside Kubernetes platform. Moreover, working with Pinot permits us to delve into each software program and infrastructure layers. We’ve developed customized Python tooling that wraps round Pinot to standardize operations, and we’ve streamlined information ingestion by way of Airflow pipelines. On the infrastructure aspect, we’ve automated deployments and upkeep duties utilizing Ansible, Kubernetes, and Jenkins.

When interviewing with Slack, my potential co-workers had been supportive and type, and this continues to be the case as we work on initiatives, deal with incidents, and plan future work. I’ve the chance to personal initiatives, work cross-functionally and broaden my talent set — all whereas working with co-workers I belief and revel in.

Hello, I’m Nilanjana Mukherjee, and I’m a Employees Software program Engineer on the Metrics Foundations workforce. I joined in October 2021.

My workforce is answerable for producing actionable datasets and metrics for data-driven decision-making. We additionally preserve excessive information accuracy and desired touchdown time SLAs. 

Becoming a member of the Knowledge Engineering workforce two years in the past has been an unbelievable expertise with alternatives and challenges. Amidst my involvement in lots of information modeling initiatives, I led a pivotal strategic initiative: a migration of workloads from Hive to Spark 3. Being one of many earliest adopters of Spark at Slack made it troublesome however rewarding. I grew to become the Spark material knowledgeable to over 40 groups throughout Slack.

The journey was difficult however considerably contributed to my private {and professional} growth. I owe a lot of my success to the assets and unwavering assist from my managers and workforce members. Lately, I used to be thrilled to be promoted from Senior Engineer to Employees Engineer a recognition of my development, and a testomony to the alternatives for development at Slack in information engineering.

Hello, I’m Ramya Sundaresan, and I’m a Senior Software program Engineer on the Metrics Foundations workforce. I joined in Could 2022.

I’ve had the chance to collaborate with two information groups throughout my tenure at Slack: the Knowledge Ingestion workforce and the Metrics Foundations workforce. On the Knowledge Ingestion workforce, we had been entrusted with the vital process of amassing, processing, and ingesting information into our information warehouse from a wide range of sources. This included extracting information from Slack’s Vitess software database, managing personalized log pipelines, and integrating information retrieval pipelines from Google Sheets. Throughout the Metrics Foundations workforce, I discovered frequent floor with Nilanjana as we centered on related areas of curiosity and experience.

My contributions to Slack’s information engineering ecosystem embody implementing Apache Iceberg inside Kafka Join clusters, orchestrating the migration of Airflow to Kubernetes from AWS EC2 (Amazon Internet Providers Elastic Compute Cloud) cases, and migrating jobs to Spark3 on AWS EMR6 (Elastic MapReduce) clusters. We closely leverage AWS. Transitioning from Spark 2 on AWS EMR 5 to Spark 3 upgrades on AWS EMR 6 was a strategic endeavor. It was according to a company-wide goal to ascertain a unified tech stack and diminish reliance on legacy techniques like Hive and MapReduce. I led a number of groups by way of automated workflows and complete documentation leading to a assured and profitable migration. We achieved our migration purpose in beneath a 12 months utilizing parallel construct pipelines and rigorous testing. This unlocked higher efficiency and fortified our techniques with improved compatibility, interoperability, and long-term assist.

My journey as a software program engineer at Slack has given me alternatives to steer, innovate, and contribute meaningfully to my workforce’s targets. Our on-call rotation and incident administration prioritizes workforce member well-being with group and automation. My work provides me loads of satisfaction, however I additionally must acknowledge a pleasant fixed in my life — my toddler son, who typically consists of the colourful Slack brand at any time when he doodles, and jogs my memory of the fun past code.

My toddler’s drawing of Mario and Luigi holding the Slack brand as an alternative of stars

Hello, I’m Shrushti Patel, and I’m a Senior Software program Engineer on the Knowledge Infrastructure workforce. I joined in August 2020.

My workforce owns all of the infrastructure and providers required to ship dependable and well timed information. Being a part of this workforce, I’ve labored on providers like AWS EMR, Airflow, Trino, Secor, and Ranger to handle infrastructure.

After I joined, Slack had an information ingestion setup reliant on Secor for transferring Kafka information to S3 hosted on EC2. This was inflicting points after we tried to include new information subjects. My workforce decided that Secor fell brief by way of industry-standard recognition, ongoing growth, and assist for rising information codecs. I assumed accountability for this Secor setup, and I spearheaded its migration to Bedrock (an internally-developed Kubernetes framework). This transition simplified the addition of latest subjects, harnessed the advantages of Kubernetes, and resulted in price financial savings by way of useful resource optimization.

To additional improve our information infrastructure, we’re migrating from Secor to Kafka Join: a widely-adopted {industry} normal for streaming techniques. Kafka Join presents out-of-the-box assist for state administration, fault tolerance, and scaling. Not like Secor, Kafka Join operates as an abstraction and leverages connectors (executable JARs) inside the Kafka-Join ecosystem. This strategic shift is transferring us in direction of real-time streaming capabilities. The work aligns with our dedication to {industry} requirements, ongoing growth, and adaptableness to evolving information codecs. 

My time with Slack’s Knowledge Infrastructure workforce has been a transformative journey. I’ve delved into mentorship, partaking initiatives, and a tradition of steady studying. Slack’s dedication to mobility has enriched my development by offering me the chance to work with all of the groups beneath information and cross-organizational groups like Cloud and Observability. After I returned from maternity depart, the workforce supported me by offering context about ongoing initiatives and onboarding me again into the stream. This expertise showcased Slack’s compassion by creating an setting the place skilled excellence is nurtured alongside empathy for private milestones.

Hello, I’m Nathalie Kaligirwa, a Senior Software program Engineer on the Metrics Platform & Governance workforce. My workforce builds scalable and standardized instruments to reinforce the information expertise. I joined in November 2021.

The previous two years have been eventful. I had a wholesome child, took six months of maternity depart, and labored on complicated initiatives with a number of stakeholders and distributed groups. I aimed to stability private development with work commitments by aligning duties with my power degree and busy schedule.

Metrics standardization is a serious ongoing initiative at Slack, and it encompasses initiatives like Merlin, a framework for abstracting metrics and extensive tables creation. Throughout parenthood, I centered on particular parts inside this initiative that might be manageable throughout a time of elevated private calls for. This allowed me to contribute to the bigger imaginative and prescient of the venture whereas balancing new father or mother obligations. For instance, I up to date a service to dynamically add new metrics created with Merlin and eradicated the necessity for guide migration. I then switched gears to work on bettering the search expertise. This set the groundwork for a brand new metadata service that gives lineage throughout a number of information instruments and scalable search.

Whether or not I used to be consolidating metrics, studying to alter diapers, integrating information providers, placing collectively a nursery, engaged on metadata providers, or navigating a compromised sleep schedule, I grew professionally and personally. Trying again, I’m happy with my work and grateful for my workforce’s assist.

Suzanna Khatchatrian, Senior Director, Knowledge Engineering. Joined October 2018
Lakshmi Mohan, Director, Knowledge Platform. Joined February 2021
Beate Porst, Group Product Supervisor, Knowledge Engineering. Joined November 2022

As Slack’s Knowledge Engineering management, we’re dedicated to mentorship and offering alternatives to study and develop. The workforce can be dedicated to mobility we encourage inside transfers for skilled development and diversification of abilities. Slack’s steady studying philosophy not solely enriches particular person careers but additionally creates a dynamic and collaborative office. 

We mannequin Slack’s worth of compassion in our assist for workforce members taking maternity depart. The workforce is organized to make reintegration after a hiatus a seamless expertise. We genuinely care not nearly our teammate’s skilled efficiency but additionally their private milestones. 

As leaders at Slack, we actively nurture an setting the place ladies in information engineering thrive and attain their full potential. We do that by way of:

  • Visibility: We amplify ladies’s voices by encouraging them to tackle management roles and communicate at conferences. We have a good time their achievements and showcase them as position fashions by way of participation in {industry} occasions.
  • Mentorship: We provide each formal and casual mentorship applications the place we join skilled ladies leaders with mentees for steerage and assist. This fosters confidence, management abilities and a robust community of friends.
  • Advocacy: We actively problem biases and advocate for honest practices in all features of our work setting.
  • Empowerment: We assist versatile work preparations, variety and inclusion initiatives, and entry to coaching assets. We create a welcoming and inclusive area for all.

Embracing variety: our dedication to inclusion

Our private experiences at Slack show the influence of those inclusion efforts. That is the Slack we envisioned: a spot the place concepts are heard, contributions are valued, and particular person journeys are fueled by a collaborative and supportive tradition. Our success is obvious within the rising variety of ladies selecting to hitch our groups and within the legacy of compassion and inclusion we’re proud to domesticate.

Involved in becoming a member of our Knowledge Engineering workforce? Apply now