Byte Journeys

Join me as I share insights and discoveries from my journey in the world as a software engineering manager by day and tinkerer by night.

Comparing AI coding models

07 February 2025

Introduction

I am relying on Ollama for my day-to-day work. I don’t want all of my data to travel outside and therefore I really appreciate this project to run LLMs locally. With a MacBook Pro M3 with 18GB RAM, this tends to run really well for models with a reasonable parameter size.

I wanted to understand how good these models are at generating code, to maybe use them as local coding assistants, so I gave them a simple task:

“Provide complete working code for a realistic looking tree in Python using the Turtle graphics library and a recursive algorithm.”

After running the code that is generated - although not very scientific - I think the quality of the code can already be judged by the output. Besides the local models, I also let the prompt run against some online models that of course have many more parameters.

Local models using Ollama

Qwen2.5-coder (1.5b)

  • Code: runs!
  • Performance: great
  • Notes: The output resembles a bush more than a tree.

Qwen2.5-coder 1.5b

import turtle

# Setup the window
screen = turtle.Screen()
turtle.bgcolor("white")
screen.title("Realistic Tree")

# Set up the turtle
pen = turtle.Turtle()
pen.speed(10)
pen.color("brown")
pen.width(2)

# Function to draw a tree trunk
def draw_trunk(length):
    if length > 5:
        pen.forward(length)
        pen.right(30)
        draw_trunk(length - 15)
        pen.left(60)
        draw_trunk(length - 15)
        pen.right(30)
        pen.backward(length)

# Function to draw a tree
def draw_tree(depth, length):
    if depth > 0:
        draw_trunk(length)
        pen.left(20)
        draw_tree(depth - 1, length * 0.8)
        pen.right(40)
        draw_tree(depth - 1, length * 0.8)
        pen.left(20)
        pen.backward(length)

# Call the function to start drawing the tree
draw_tree(5, 100)

# Keep the window open until manually closed
screen.mainloop()

Qwen2.5-coder (3b)

  • Code: runs!
  • Performance: great
  • Notes: Why is the tree on its side? Also, this took forever to draw with the parameters the AI chose, so the output is very detailed. What’s up with the green background? It generated some nicely commented code.

Qwen2.5-coder 3b

import turtle

def draw_tree(turtle, length, angle):
    if length > 5:
        # Draw the trunk of the tree
        turtle.forward(length)

        # Turn left to make a branch
        turtle.left(angle)
        draw_tree(turtle, length * 0.8, angle - 10)  # Recursive call for the left branch

        # Move back to the root of the current subtree
        turtle.right(2 * angle)

        # Draw the other branch
        draw_tree(turtle, length * 0.8, angle - 10)  # Recursive call for the right branch

        # Turn back to face the original direction
        turtle.left(angle)

        # Move back to the root of the current subtree again
        turtle.backward(length)

def main():
    # Set up the turtle screen
    window = turtle.Screen()
    window.bgcolor("green")

    # Create a Turtle object
    tree_turtle = turtle.Turtle()
    tree_turtle.speed(0)  # Set the speed to maximum

    # Start drawing the tree
    draw_tree(tree_turtle, 100, 45)

    # Hide the turtle and keep the window open
    tree_turtle.hideturtle()
    window.mainloop()

if __name__ == "__main__":
    main()

Qwen2.5-coder (7b)

  • Code: runs!
  • Performance: great
  • Notes: Ok, it got the color of a tree more or less right, but… that’s really not a tree. But I like that it introduced some randomness into the whole thing.

Qwen2.5-coder 7b

import turtle
import random

# Set up the screen
screen = turtle.Screen()
screen.bgcolor("black")
pen = turtle.Turtle()

# Function to draw a tree recursively
def draw_tree(branch_len, t):
    if branch_len > 5:
        # Draw branch down
        t.forward(branch_len)
        t.color("saddlebrown")
        angle = random.randint(15, 40)  # Random angle for branches
        shorten_by = random.uniform(0.7, 0.8)  # Random shortening factor
        t.right(angle)
        draw_tree(branch_len * shorten_by, t)

        # Draw branch left
        t.left(2 * angle + 5)
        draw_tree(branch_len * shorten_by, t)

        # Return to the starting position
        t.right(angle)
        t.backward(branch_len)
        t.color("green")  # Reset color to green for leaves

# Set up initial conditions
pen.speed(0)
pen.left(90)  # Starting angle
pen.up()
pen.backward(150)  # Move the starting position down
pen.down()

# Draw the tree
draw_tree(80, pen)

# Hide the turtle and display the result
pen.hideturtle()
turtle.done()

Qwen2.5-coder (14b)

  • Code: runs!
  • Performance: good
  • Notes: This is pretty compact code that got generated. I like the comments in the code. Why is the tree on its side again and in the lower right corner of the output? I think this is a great example where there is compact code, that generates compact output.

Qwen2.5-coder 14b

import turtle

# Set up the screen
screen = turtle.Screen()
screen.bgcolor("white")

# Create a turtle named 'tree'
tree = turtle.Turtle()
tree.speed(0)  # Fastest drawing speed
tree.color("brown")
tree.penup()
tree.goto(0, -200)  # Move the turtle to the base of the tree
tree.pendown()

# Function to draw the tree recursively
def draw_tree(branch_length, t):
    if branch_length > 5:
        t.forward(branch_length)
        t.right(20)  # Turn right by 20 degrees
        draw_tree(branch_length - 15, t)  # Recursive call for the right subtree
        t.left(40)  # Turn left by 40 degrees to create a split
        draw_tree(branch_length - 15, t)  # Recursive call for the left subtree
        t.right(20)  # Turn right back to original direction
        t.backward(branch_length)  # Move backward to return to previous position

# Draw the tree with an initial branch length of 100
draw_tree(100, tree)

# Hide the turtle and display the window
tree.hideturtle()
turtle.done()

Llama3.3 (70b)

70b parameters is too large for my machine and thus also not usable as a local coding assistant. It was running for over an hour and only produced:

import turtle

Llama3.2 (3b)

  • Code: runs!
  • Performance: great
  • Notes: I like the color, I like the tree, but why is it again on its side? As for the code: I liked how the code contained a description of the function’s parameters.

Llama 3.2 3b

import turtle

def draw_tree(t, branch_length, angle, shrink_factor, min_length):
    """
    Draw a tree by recursively drawing branches.

    Args:
        t (turtle.Turtle): The turtle object used for drawing.
        branch_length (float): The length of the current branch.
        angle (float): The angle at which the next branch is drawn.
        shrink_factor (float): The factor by which each branch is reduced in size.
        min_length (float): The minimum length of a branch before it stops being drawn.
    """
    if branch_length < min_length:
        return

    t.forward(branch_length)
    t.left(angle)
    draw_tree(t, branch_length * shrink_factor, angle, shrink_factor, min_length)
    t.right(2 * angle)
    draw_tree(t, branch_length * shrink_factor, angle, shrink_factor, min_length)
    t.left(angle)
    t.backward(branch_length)

def main():
    window = turtle.Screen()
    window.bgcolor("white")

    my_turtle = turtle.Turtle()
    my_turtle.speed(0)  # fastest speed
    my_turtle.color("green")  # green color

    draw_tree(my_turtle, 100, 30, 0.7, 5)

    window.mainloop()

main()

Mistral (7b)

  • Code: runs!
  • Performance: great
  • Notes: Yeah, the color is green, but that is not a tree. The code looks like someone who just got into coding did it.

Mistral 7b

import turtle

# Define constants for parameters
TRUNK_LENGTH = 100
BRANCH_LENGTH = 80
LEAF_LENGTH = 20
ANGLE = 35

def draw_branch(length):
    if length > 4:
        turtle.forward(length)
        turtle.right(ANGLE)
        draw_branch(length - 20)
        turtle.left(2 * ANGLE)
        draw_branch(length - 20)
        turtle.backward(length)
        turtle.left(ANGLE)

def draw_leaf():
    for _ in range(3):
        turtle.forward(LEAF_LENGTH)
        turtle.right(120)
        turtle.forward(LEAF_LENGTH)
        turtle.left(120)

def draw_tree(length=TRUNK_LENGTH):
    turtle.penup()
    turtle.goto(-150, TRUNK_LENGTH / 2)
    turtle.pendown()
    turtle.pencolor("brown")
    turtle.fillcolor("green")
    turtle.begin_fill()
    turtle.right(45)
    turtle.forward(length)
    turtle.right(-90)
    draw_branch(BRANCH_LENGTH)
    turtle.end_fill()

def main():
    turtle.bgcolor("skyblue")
    turtle.speed(1)
    draw_tree()
    turtle.hideturtle()
    turtle.done()

if __name__ == "__main__":
    main()

Codestral (22b)

  • Code: runs!
  • Performance: poor
  • Notes: This took a long while to generate as it’s a quiet big model for my computer. But the tree looks good and it’s upright! The code is nicely commented and pretty lean, especially compared to the previous code from the Mistral model.

Codestral 22b

import turtle

# Function to draw a tree
def draw_tree(length, angle):
    if length > 10:
        # Move the turtle forward by specified length
        turtle.forward(length)
        # Turn the turtle right by specified angle
        turtle.right(angle)

        # Recursive call for the right sub-tree
        draw_tree(0.7 * length, angle)

        # Turn the turtle left by 2 times the specified angle
        turtle.left(2*angle)

        # Recursive call for the left sub-tree
        draw_tree(0.7 * length, angle)

        # Turn the turtle right by specified angle to return to original direction
        turtle.right(angle)
        # Move the turtle backward by specified length
        turtle.backward(length)

# Set up the screen and turtle for drawing
turtle.speed('fastest')  # Set the speed of drawing to fastest
turtle.left(90)          # Turn turtle left by 90 degrees
draw_tree(100, 30)       # Start drawing tree with initial length and angle
turtle.done()            # Keeps the window open until it's manually closed

Gemma2 (2b)

  • Code: runs!
  • Performance: great
  • Notes: Yeah, there is no tree. Thanks for the effort. At least it changed the title of the window. The code is pretty verbose I think.

Gemma2 2b

import turtle
import random

# Screen setup
screen = turtle.Screen()
screen.setup(500, 500)
screen.title("Realistic Tree")
screen.tracer(0)  # Disable animation for faster drawing

# Turtle setup
pen = turtle.Turtle()
pen.speed(1) # Set turtle speed to slow
pen.hideturtle()
pen.penup()
pen.color("brown")
pen.shape("turtle")  # Optional: Change pen shape 

def draw_branch(angle, length, depth):
  """Recursive function for drawing branches."""
  if depth <= 0:  
    return
  pen.pendown()
  pen.forward(length)
  pen.left(angle)
  draw_branch(angle-15, length*random.uniform(0.75,1), depth - 1)  # Adjust angle and random length for variation

def draw_leaves(depth):
  """Recursive function for drawing leaves."""
  if depth <= 0: 
    return
  pen.penup()
  pen.color("green")  
  pen.forward(50) 
  pen.left(180)
  pen.pendown()

  # Repeat the process with smaller angles and depths to create a tree-like structure
  draw_leaves(depth-1)


# Main function
def draw_tree():
  """Generates a tree using recursion."""
  # Draw the trunk (add your own details for more realism, like trunk color, bark texture)
  pen.pendown() 
  pen.color("brown")
  pen.begin_fill()
  pen.goto(0, -100) # Place trunk here

  pen.end_fill()
  draw_leaves(5)
  # Draw the branches (recursive calls for variation in size and angle)
  draw_branch(90, 200, 1)


turtle.done()

Gemma2 (9b)

  • Code: runs!
  • Performance: great
  • Notes: Finally we get a tree from Gemma2. But again it’s in the bottom right. And it’s also on its side again.

Gemma2 9b

import turtle
import random

# Screen setup
screen = turtle.Screen()
screen.bgcolor("lightgreen")  

# Turtle setup
pen = turtle.Turtle()
pen.speed(0)  
pen.hideturtle()
pen.penup()

# Tree drawing function (recursive)
def draw_tree(branch_len, angle):
    if branch_len > 5:
        pen.pendown()
        pen.forward(branch_len)
        
        # Randomly vary branch thickness for realism
        pen.pensize(random.randint(1, 3))
        
        # Draw branches recursively
        pen.left(angle)
        draw_tree(branch_len - 10, angle - 5)
        pen.right(2 * angle)
        draw_tree(branch_len - 10, angle - 5)
        pen.left(angle)
        
        pen.penup()
        pen.backward(branch_len)
        pen.pendown()

# Starting tree drawing
pen.goto(0, -200)  
draw_tree(100, 30)

# Keep the window open until it's closed manually
turtle.done()

Gemma2 (27b)

Here again, my hardware is not large enough to run this in sufficient time.

Deepseeker-r1 (7b)

The performance was great, it produced some code, but the code did not run. The code seems to be… a lot.

import turtle
import random

def draw_tree(turtle_obj, x, y, length=100, angle_range=(25, 45)):
    # Move the turtle to the starting position
    turtle_obj.penup()
    turtle_obj.goto(x, y)
    turtle_obj.setheading(90)  # Point upwards initially

    # Draw the trunk (first line segment)
    turtle_obj.pendown()
    turtle_obj.width = 15  # Width of the trunk
    turtle_obj.forward(length)

    # After drawing the trunk, move to the end of the trunk
    last_point = turtle_obj.position()
    turtle_obj.trail()  # Set the trail to track drawn lines

    def draw_branch(turtle_obj, x, y, length):
        if length < 10:
            return  # Base case: stop when branch is too small

        # Calculate random angle within range
        angle = random.uniform(angle_range[0], angle_range[1])

        turtle_obj.penup()
        turtle_obj.goto(x, y)

        # Move forward to the next position
        turtle_obj.forward(length)

        # Turn by random angle and draw a small circle (representing leaves)
        turtle_obj.setheading(90 - angle + 180)  # Set direction before drawing leaves
        turtle_obj.width = length * 0.5  # Smaller width for branches

        # Draw left branch recursively
        draw_branch(turtle_obj, x, y, int(length * 0.7))

        # Draw right branch recursively (with a different angle)
        angle2 = random.uniform(angle_range[0], angle_range[1])
        turtle_obj.setheading(90 - angle2 + 180)
        draw_branch(turtle_obj, x, y, int(length * 0.7))

        # Move back to the starting position after drawing both branches
        turtle_obj.pendown()
        distance = abs(turtle_obj.xcor() - last_point.x) + abs(turtle_obj.ycor() - last_point.y)
        turtle_obj.setheading(last_point[0] - y, last_point[1] - x)
        if turtle_obj.distance(last_point) > 3:
            turtle_obj.move(distance * (length / distance))

    # Draw the tree
    draw_branch(turtle_obj, last_point.x + (last_point.y + length), -length)

    # Cleanup: hide trails and reset settings
    turtle_obj.trail = None
    turtle_obj hideturtle()
    turtle_obj.reset heading(90)
    turtle_obj.speed(None)  # Optimize speed after drawing

# Setup the window
turt = turtle.Turtle()
window = turtle.Screen()

def main():
    global turt

    turt.speed('slowabs')  # Slow speed for better visibility

    # Initial parameters: starting position, trunk height, and width
    turt.setworldcoordinates(-100, -40, 350, 270)
    turt.penup()
    turt.goto(150, -120)  # Start at the bottom center

    draw_tree(turt, 150, -120, angle_range=(25, 35), length=60)

    window.setup(width=window.width(), height=window.height())
    window.update()

if __name__ == '__main__':
    main()

Remote models

I then moved on to try out the remote models.

OpenAI GPT-4o

  • Code: runs!
  • Performance: great
  • Notes: This tree is really nice looking. But I really am not a fan of the color scheme that this model chose. The code looks pretty straight forward. It’s the first time that I saw that the program has a defined exit (when clicked on the window). That’s a nice touch.

OpenAI GPT-4o

import turtle
import random

def tree(branch_length, t):
    if branch_length > 5:
        # Randomize the size of branches and angles to make the tree more realistic
        t.forward(branch_length)
        angle = random.randint(15, 45)
        t.right(angle)
        tree(branch_length - random.randint(10, 20), t)
        t.left(angle * 2)
        tree(branch_length - random.randint(10, 20), t)
        t.right(angle)
        t.backward(branch_length)

def main():
    # Set up the screen
    window = turtle.Screen()
    window.bgcolor("blue")

    # Create turtle
    t = turtle.Turtle()
    t.speed(0)  # Fastest drawing speed
    t.color("green")
    t.left(90)       # Point turtle to go upwards
    t.up()
    t.backward(100)  # Move turtle a bit downwards to center the tree
    t.down()
    t.pensize(2)

    # Drawing the tree
    tree(100, t)

    # Click on screen to exit
    window.exitonclick()

if __name__ == "__main__":
    main()

OpenAI GPT-3o-mini

  • Code: runs!
  • Performance: great
  • Notes: The tree looks nice, the code also looks like it’s pretty straightforward. As a PR commenter I would have asked to move code away from main() some more.

OpenAI GPT-3o-mini

import turtle
import random

def draw_tree(branch_length, t):
    if branch_length > 5:
        t.forward(branch_length)
        angle = random.randint(15, 45)
        reduction = random.uniform(0.6, 0.8)

        t.left(angle)
        draw_tree(branch_length * reduction, t)
        t.right(2 * angle)
        draw_tree(branch_length * reduction, t)
        t.left(angle)
        t.backward(branch_length)

def main():
    window = turtle.Screen()
    window.bgcolor("lightblue")
    t = turtle.Turtle()
    t.left(90)
    t.up()
    t.backward(100)
    t.down()
    t.speed("fastest")
    t.color("brown")

    draw_tree(100, t)

    window.mainloop()

main()

Anthropic Claude 3.5 Sonnet

  • Code: runs!
  • Performance: great
  • Notes: The brush that was taken to draw this tree is pretty big. Also, the program runs for a really long time, but somehow the tree feels fluffy. The code feels pretty verbose.

OpenAI GPT-3o-mini

import turtle
import random

def draw_tree(t, branch_length, angle, depth):
    if depth == 0:
        return

    # Draw the main branch
    t.forward(branch_length)

    # Save the current position
    current_pos = t.pos()
    current_heading = t.heading()

    # Right sub-branch
    t.right(angle)
    draw_tree(t, branch_length * 0.8, angle * 0.9, depth - 1)

    # Return to the saved position
    t.penup()
    t.setpos(current_pos)
    t.setheading(current_heading)
    t.pendown()

    # Left sub-branch
    t.left(angle)
    draw_tree(t, branch_length * 0.8, angle * 0.9, depth - 1)

    # Return to the saved position
    t.penup()
    t.setpos(current_pos)
    t.setheading(current_heading)
    t.pendown()

    # Middle sub-branch (adds more realism)
    t.left(angle / 2)
    draw_tree(t, branch_length * 0.8, angle * 0.9, depth - 1)

    # Return to the start of this branch
    t.penup()
    t.setpos(current_pos)
    t.setheading(current_heading)
    t.pendown()

def main():
    # Set up the screen
    screen = turtle.Screen()
    screen.setup(800, 600)
    screen.bgcolor("skyblue")

    # Create the turtle
    t = turtle.Turtle()
    t.speed(0)  # Fastest drawing speed
    t.hideturtle()

    # Set initial position
    t.penup()
    t.setpos(0, -250)
    t.setheading(90)
    t.pendown()

    # Set color and pen size
    t.color("brown")
    t.pensize(10)

    # Draw the tree
    draw_tree(t, 120, 30, 8)

    # Keep the window open
    screen.mainloop()

if __name__ == "__main__":
    main()

GitHub Copilot

  • Code: runs!
  • Performance: great
  • Notes: Ok, so this is a tree! It’s brown, it has a trunk that is larger than the branches! The tress that I can generate with this all look different, but they all look great. Also, when I click on the window, it closes and the program exits!

GitHub Copilot

import turtle
import random

def draw_branch(branch_length, t):
    if branch_length > 5:
        # Adjust the thickness of the branches
        t.width(branch_length // 10)

        # Draw the main branch
        t.forward(branch_length)

        # Draw the right branch
        angle = random.randint(15, 45)
        t.right(angle)
        draw_branch(branch_length - random.randint(10, 20), t)

        # Draw the left branch
        t.left(2 * angle)
        draw_branch(branch_length - random.randint(10, 20), t)

        # Return to the original position and angle
        t.right(angle)
        t.backward(branch_length)

def main():
    screen = turtle.Screen()
    screen.bgcolor("sky blue")

    t = turtle.Turtle()
    t.color("saddle brown")
    t.speed(0)
    t.left(90)
    t.up()
    t.backward(100)
    t.down()

    draw_branch(100, t)

    screen.exitonclick()

if __name__ == "__main__":
    main()

Conclusion

This was a fun exercise. It’s not very scientific, but nevertheless I think it gives some value. Obivously GitHub Copilot is the overall clear winner and there is no doubt why engineers like to work with it so much.

As for the local models, I think codestral with 22b parameters could work well on a little bit beefier machine to aid with local coding.