Bringing Rigor to Synapse: Static Code Analysis via Azure DevOps

Ensuring code quality is paramount, and for Azure Synapse users, this just got a lot easier.

Azure Synapse Analytics is an integrated analytics service that accelerates time to insight from all data at any scale, across data warehouses and big data analytics. But, even the most robust platforms can have gaps — for Synapse, it’s the integration of popular code quality tools and IDE support. That’s where this solution comes into play.

The Challenge

For Synapse aficionados, you’ve likely felt the absence of something like SonarQube integration or VSCode assistance in your Synapse Studio environment. The native browser IDE is great, but when it comes to writing code that’s not just good but excellent, a little help goes a long way.

The Solution

Enter Azure DevOps. The pipeline I’m about to detail will be your new sidekick in maintaining high standards of code quality for Synapse. Let’s break down how it works.

Install Dependencies

We begin with setting the stage for our pipeline by installing all necessary Python packages.

- script: |
    echo "Installing dependencies..."
    pip install synapse-to-ipynb nbqa pylint flake8
  displayName: 'Install Dependencies'

Convert Synapse Notebooks from json to ipynb

Synapse notebooks in git are all json files and not that useful. To perform static code analysis, we convert them into Jupyter Notebooks (ipynb). This step does exactly that.

- script: |
    echo "Converting Synapse notebooks to Python scripts..."
    mkdir python_notebooks
    python /home/AzDevOps/.local/bin/synapse-to-ipynb --source notebook/ --target python_notebooks/
  displayName: 'Convert Synapse Notebooks'

Code Analysis on Notebooks

With our notebooks converted, we then move on to the meat of the process: enforcing style guides and performing static code analysis.

- script: |
    find python_notebooks/ -type f -name "*.ipynb" | while read notebook; do
      echo "Check for style guide enforcement with flake8 on $notebook"
      python /home/AzDevOps/.local/bin/nbqa flake8 "$notebook"

      echo "Perform static code analysis with pylint on $notebook"
      python /home/AzDevOps/.local/bin/nbqa pylint "$notebook"
    done
  displayName: 'Code analysis on notebooks'

Tooling Explained

• nbqa: A handy tool that allows quality checks on Jupyter Notebooks.
• pylint: This is your enforcer for coding standards.
• flake8: Think of it as the guardian of code complexity and style.

In action

And how does this look when the rubber meets the road? Below is a screenshot from a real pipeline run, showing the kind of output you can expect:

How you can use this

To implement this pipeline in your projects, simply add the YAML code into your Azure DevOps pipeline configuration. From there, it will automatically execute these steps, providing you with a detailed analysis of your Synapse notebooks.

Remember, maintaining high standards in your code ensures fewer bugs, better maintainability, and a smoother development cycle. With this pipeline, you’re not just coding; you’re crafting quality into the very fabric of your Synapse projects.