In a world where the post-pandemic dust is beginning to settle, one thing has become increasingly clear: Data teams are in high demand.

The driving force behind this demand is not just hype but the tangible impact of large language models or LLMs and the challenges of effectively using them — especially as LLMs become SLMs, or specialized language models, where organizations are building their own language models trained on their proprietary data, their bench strength.

In this article, we delve into the evolving state of tech recruiting for data teams. To provide insights into this dynamic field, we brought in Alex Hutchins, co-founder of DataWorks, a global recruiting firm specializing in helping organizations build their data teams.

The state of tech recruiting

As we navigate through the ever-changing landscape of tech recruiting, it’s evident that the market has experienced significant shifts. The past year has been a rollercoaster ride, from a frenzied hiring spree to a sudden slowdown, all while companies grapple with economic uncertainties.

In our conversation, Alex reflects on this journey, starting from the early days of the pandemic when companies were aggressively investing and expanding their data and tech teams. However, as 2021 progressed, signs of a slowdown emerged, with offers being rescinded and requisitions disappearing from pipelines.

By the time January rolled around, tech giants, often referred to as “FAANG,” announced significant layoffs, sending shockwaves through the industry. The fallout continued, with many firms resorting to layoffs and a climate of uncertainty casting a shadow over the tech job market.

Yet we see a potential light at the end of the tunnel. The larger players in the industry are showing signs of rehiring, although it’s unlikely we will witness a return to the pre-pandemic hiring frenzy. Companies have had to reckon with factors such as inflation, repaying COVID loans, geopolitical issues and the role of artificial intelligence in their strategies, which has led to a more cautious approach to hiring.

Recruiting for data teams

One of the most intriguing aspects of the tech recruiting landscape is the state of hiring within data teams. Data-driven decision-making has become crucial for companies across sectors, leading to a surge in demand for data professionals. However, this demand has evolved, focusing on specialized skill sets.

We highlight the shifting dynamics within data teams. Early in the year, there was a notable reduction in middle management positions while the focus shifted to hands-on engineers and individual contributors. Companies, both startups and established players, were reevaluating their data strategies, often adopting modern data tools and re-platforming their data infrastructure.

A significant development in the field is the emergence of machine learning engineers with expertise in fine-tuning LLMs. These specialists are in high demand, reflecting the growing interest in leveraging LLMs for various applications. Additionally, data science roles have evolved, emphasizing deep domain expertise and industry-specific knowledge, which has placed new demands on data professionals.  This goes back to the trend of SLMs that are “internal LLMs” to an organization.

What we’re hearing from customers

To gain further insights into the tech recruiting landscape, it’s essential to listen to the experiences and perspectives of customers. Customers seeking to build or expand their data teams offer valuable insights into the evolving job market.

One notable trend is the bottleneck in the hiring process. With an abundance of job applications and a shortage of talent acquisition professionals, the hiring process has become more protracted. What used to take an average of 28 days from resume submission to offer has nearly doubled in duration, illustrating the challenges organizations face in filling data roles efficiently.

Another noteworthy aspect is the polarization of companies. Some organizations with strong brands and streamlined recruitment processes continue to attract top talent with relative ease. Conversely, companies inundated with applications that do not align with their needs create a subpar candidate experience.

In-demand skill sets for modern data teams

The skill sets in demand within the tech industry, particularly in data-related roles, are evolving rapidly. Understanding these skill requirements is crucial for both job seekers and organizations looking to hire top talent.

Among the most sought-after roles are machine learning engineers, focusing on fine-tuning LLMs. These specialists are pivotal in making LLMs more effective for specific data products. Additionally, data science roles have become more specialized, with companies looking for candidates with deep domain knowledge and expertise in areas such as natural language processing or NLP, computer vision and AI.

Data engineering roles remain vital, with a demand for analytics engineers proficient in tools such as dbt for modeling and cloud-based data solutions, such as Databricks or Apache Spark. Infrastructure-focused positions such as DataOps and MLOps engineers are on the rise as companies prioritize efficient data operations. Furthermore, AI product managers are gaining prominence, particularly in startups, as organizations seek professionals who can shape AI product roadmaps while understanding data and AI fundamentals.

Software engineers with full-stack experience, capable of handling both data and application development, remain highly coveted. These professionals bridge the gap between data and product development, facilitating the creation of data-driven applications.

One place we are not seeing as much demand is for data product managers. We believe this remains a critical role for a chief data officer’s success, but we still see very few openings for this role. It could be that data developers, engineers, or scientists are being forced to provide the skills and work. In the long run this could lead to data products being more data features than true products, with data product sprawl.

The road ahead

Looking ahead, the data team recruiting landscape is expected to gain more confidence as we approach the end of the year and move into 2024. Budgets are being allocated for growth, and signs indicate increased hiring activity. However, this growth is expected to be more measured, avoiding the excesses seen in the past.

The role of AI will continue to be pivotal in recruitment, with companies incorporating LLMs and AI-as-a-service solutions into their strategies. Venture capital activity will likely gain momentum, but investors will adopt a more discerning approach, focusing on startups that demonstrate genuine value and potential.

The market will see a consolidation of less specialized data roles driven by automation and AI, which will displace certain positions. Companies will continue to adapt, evolve and innovate to stay competitive in the data-driven world.

Our ANGLE: Navigating the evolving landscape

In conclusion, recruiting data teams is undergoing a significant transformation. From the uncertainties of the early pandemic days to the current resurgence in confidence, the journey has been challenging yet enlightening.

As the demand for specialized skill sets grows, job seekers and organizations must stay attuned to the evolving requirements of the tech job market. By understanding the changing landscape and adapting accordingly, they can thrive in this dynamic industry.

For those seeking more information about DataWorks and their insights into tech recruiting, reaching out to their team can provide valuable guidance and expertise in navigating this evolving landscape. It can be reached on its website or via LinkedIn.

Here’s the full video:

Photo: SiliconANGLE