Scripting API js performance

I have some javascript i wrote to load a map created with azgaar fantasy map creator into tiled. When i run the code using nodejs in vs code it takes about 47 seconds to iterate the grid and find all the closest cells (via a quadtree). The same code when run in tiled runs forever, like 20-25 minutes, if it finishes. Tiled will sometimes crash.

What can i do to create a better performing plugin? I’ve noticed a couple of references to python for extensions, but i don’t really see any examples of making python plugins. What are my options for speed?

The test is below. quadtree is a javascript class i wrote (ported from some of my c# code). If i remove the “getClosest” method from the tiled plugin, everything runs in < 50ms, but with that, it takes 1400+ seconds.

map.json is any exported json file from Azgaar's Fantasy Map Generator

import { Quadtree, Rectangle } from './quadtree.mjs';
import { readFileSync } from 'fs';
var startTime = Date.now();
function logWithTime(message)
{
    var elapsed = ((Date.now() - startTime) / 1000).toFixed(3);
    console.log("[" + elapsed + "s] " + message);
}
const data = JSON.parse(readFileSync('./map.json', 'utf-8'));

const qt = new Quadtree(0, new Rectangle(0, 0, data.info.width, data.info.height));
    

logWithTime('Created quadtree');
for (var cellIndex in data.pack.cells)
{
    var cell = data.pack.cells[cellIndex];
    qt.insert({ coordinates: { x: cell.p[0], y: cell.p[1] }, data: cell });
}

logWithTime('Inserted cells into quadtree');

for(let x = 0; x < data.info.width; x++)
{
    for(let y = 0; y < data.info.height; y++)
    {
        try
        {
            var mapX = (x + 0.5);
            var mapY = (y + 0.5);
            var cell = qt.getClosest({ x: mapX, y: mapY });
        }
        catch (e)
        {
            console.error(e);
        }
    }
}

logWithTime('Done');

class Quadtree
{
    constructor(level, bounds, maxObjects = 4, maxLevels = 5)
    {
        this.maxObjects = maxObjects;
        this.maxLevels = maxLevels;
        this.level = level;
        this.bounds = bounds;
        this.objects = [];
        this.nodes = [null, null, null, null];
        this.parent = null;
    }

    clear()
    {
        this.objects = [];
        for (let i = 0; i < this.nodes.length; i++)
        {
            if (this.nodes[i] !== null)
            {
                this.nodes[i].clear();
                this.nodes[i] = null;
            }
        }
    }

    insert(obj, addToParent = true)
    {
        if (this.nodes[0] !== null)
        {
            const index = this.getIndex(obj.coordinates);
            if (index !== -1)
            {
                this.nodes[index].insert(obj);
                return;
            }
        }

        this.objects.push({ point: obj.coordinates, data: obj });

        if (this.objects.length > this.maxObjects && this.level < this.maxLevels)
        {
            if (this.nodes[0] === null)
            {
                this.split();
            }

            for (const o of [...this.objects])
            {
                const index = this.getIndex(o.point);
                if (index !== -1)
                {
                    this.nodes[index].insert(o.data, false);
                }
            }
        }

        if (addToParent)
        {
            let node = this.parent;
            while (node !== null)
            {
                node.objects.push({ point: obj.coordinates, data: obj });
                node = node.parent;
            }
        }
    }

    retrieve(returnObjects, point)
    {
        if (!returnObjects) 
        {
            returnObjects = [];
        }
        const index = this.getIndex(point);
        if (index !== -1 && this.nodes[0] !== null)
        {
            this.nodes[index].retrieve(returnObjects, point);
        }

        returnObjects.push(...this.objects.map(o => o.data));
        return returnObjects;
    }

    getNode(point)
    {
        const index = this.getIndex(point);
        if (index !== -1 && this.nodes[0] !== null)
        {
            return this.nodes[index].getNode(point);
        }
        return this;
    }

    split()
    {
        const subWidth = this.bounds.width / 2;
        const subHeight = this.bounds.height / 2;
        const x = this.bounds.x;
        const y = this.bounds.y;

        this.nodes[0] = new Quadtree(this.level + 1, new Rectangle(x + subWidth, y, subWidth, subHeight), this.maxObjects, this.maxLevels);
        this.nodes[0].parent = this;

        this.nodes[1] = new Quadtree(this.level + 1, new Rectangle(x, y, subWidth, subHeight), this.maxObjects, this.maxLevels);
        this.nodes[1].parent = this;

        this.nodes[2] = new Quadtree(this.level + 1, new Rectangle(x, y + subHeight, subWidth, subHeight), this.maxObjects, this.maxLevels);
        this.nodes[2].parent = this;

        this.nodes[3] = new Quadtree(this.level + 1, new Rectangle(x + subWidth, y + subHeight, subWidth, subHeight), this.maxObjects, this.maxLevels);
        this.nodes[3].parent = this;
    }

    getIndex(point)
    {
        let index = -1;
        const verticalMidpoint = this.bounds.x + (this.bounds.width / 2);
        const horizontalMidpoint = this.bounds.y + (this.bounds.height / 2);

        const topQuadrant = (point.y < horizontalMidpoint && point.y >= this.bounds.y);
        const bottomQuadrant = (point.y >= horizontalMidpoint);

        if (point.x < verticalMidpoint && point.x >= this.bounds.x)
        {
            if (topQuadrant)
            {
                index = 1;
            } else if (bottomQuadrant)
            {
                index = 2;
            }
        } else if (point.x >= verticalMidpoint)
        {
            if (topQuadrant)
            {
                index = 0;
            } else if (bottomQuadrant)
            {
                index = 3;
            }
        }

        return index;
    }

    getObjects(point)
    {
        let node = this.getNode(point);

        if (node.parent === null)
        {
            return node.objects.map(x => x.data);
        }

        while (node.parent !== null)
        {
            const r = new Rectangle(
                Math.max(0, Math.min(node.bounds.width - (node.bounds.width / 2), node.bounds.width)),
                Math.max(0, Math.min(node.bounds.height - (node.bounds.height / 2), node.bounds.height)),
                node.bounds.width,
                node.bounds.height
            );

            node = node.parent;

            if (node.nodes.every(x => x && x.bounds.intersects(r)))
            {
                return node.objects.map(x => x.data);
            }
        }

        return [];
    }

    getClosest(point)
    {
        const objects = this.getObjects(point);

        if (objects.length === 0) return null;

        return objects.reduce((closest, obj) =>
        {
            const dist = this.distance(obj.coordinates, point);
            const closestDist = this.distance(closest.coordinates, point);
            return dist < closestDist ? obj : closest;
        });
    }

    distance(p1, p2)
    {
        const dx = p2.x - p1.x;
        const dy = p2.y - p1.y;
        return Math.sqrt(dx * dx + dy * dy);
    }
}

class Rectangle
{
    constructor(x, y, width, height)
    {
        this.x = x;
        this.y = y;
        this.width = width;
        this.height = height;
    }

    contains(point)
    {
        return (point.x >= this.x && point.x <= this.x + this.width &&
            point.y >= this.y && point.y <= this.y + this.height);
    }

    intersects(range)
    {
        return !(range.x > this.x + this.width ||
            range.x + range.width < this.x ||
            range.y > this.y + this.height ||
            range.y + range.height < this.y);
    }
}

export { Quadtree, Rectangle };

Qt has a custom JS engine which is more optimized for executing smaller snippets of JS than the engine used by NodeJS. Its overhead appears to be rather significant in your computationally heavy script. Since you’re not actually using the Tiled JS API, it looks like you’re hit by purely JS-engine related slowness.

That said, if the quad tree implementation was correct it should have no problem running in reasonable time also within Tiled. But:

  • getObjects looks broken (it effectively either returns the root objects or the objects in a leaf’s parent).
  • In general there are a lot of unnecessary allocations being done (getObjects returns copy, though no copy should be needed for getClosest).
  • The distance to the closest-object-so-far is re-calculated repeatedly.
  • You don’t need Math.sqrt when looking for the nearest object.
  • Probably you should just implement getClosest efficiently without relying on getObjects.
  • Why does a node still store any objects even after it has been split? The Quadtree would be simpler (and take less memory) when only leaf nodes would store objects.
  • Why are the objects wrapped in a { point: obj.coordinates, data: obj }? In the insert function the Quadtree already assumes the objects have a coordinates member, so it could just use it instead of point.

I’ve addressed the above in the following implementation of getClosest:

    getNodeWithObjects(point)
    {
        // If we're a non-empty leaf, it's us
        if (this.objects)
        	return this.objects.length > 0 ? this : null;

        // Otherwise, check the closest child node
        const index = this.getIndex(point);
        const node = this.nodes[index].getNodeWithObjects(point);
        if (node)
            return node;

        // If it was empty, return any non-empty child
        // (there has to be one otherwise we'd still be a leaf)
        for (let i = 0; i < this.nodes.length; i++)
        {
            if (i === index) // this one was empty
                continue;
            const node = this.nodes[i].getNodeWithObjects(point);
            if (node)
                return node;
        }

        return null;
    }


    getClosest(point) {
        let closest = null;
        let closestDistanceSq = Infinity;
        let searchRect = null;

        let startNode = this.getNodeWithObjects(point);
        if (!startNode)
            return null;

        const considerNode = (node) => {
            for (const w of node.objects) {
                const d2 = this.distanceSquared(w.coordinates, point);
                if (d2 < closestDistanceSq) {
                    closestDistanceSq = d2;
                    closest = w;

                    const d = Math.sqrt(d2);
                    searchRect = new Rectangle(point.x - d, point.y - d, d * 2, d * 2);
                }
            }
        };

        considerNode(startNode);

        // Iterate all leaf nodes intersecting with searchRect
        (function visit(n) {
            if (n == startNode || !n.bounds.intersects(searchRect))
                return;

            if (!n.nodes)
                considerNode(n);
            else
                n.nodes.forEach(visit);
        })(this);

        return closest;
    }

    distanceSquared(p1, p2)
    {
        const dx = p2.x - p1.x;
        const dy = p2.y - p1.y;
        return dx * dx + dy * dy;
    }

The idea is:

  • First we find the smallest node covering the requested location that still contains any objects. I introduced getNodeWithObjects for this.
  • When there is such a node, we determine the closest object.
  • From the distance to the closest object, we determine a rectangle within which there might be closer objects.
  • Then we iterate the quad tree recursively, only going down into nodes which intersect with our search rectangle, which gets smaller each time we find a closer object.

Note that I initialized nodes to null rather than [null, null, null, null], which simplifies checking whether we have any child nodes (and avoiding further allocations).

For my exported JSON file, with data.info indicating a size of 1994x1066 and 5447 cells, node can perform the script in about 2 seconds. Unfortunately Tiled is still much slower, but is now able to run this script in about 2 minutes. That’s still slower than the time it took nodejs on the original version, but I hope it might be fast enough.

Note that I did not check the script for correctness!

Here’s the full version of the adjusted script (needs rename to .mjs, but forum only allows .js for now): quadtree.js (5.8 KB)

Edit: My initial getNodeWithObjects returned null in case the point was in an empty leaf, I’ve fixed this now.

1 Like

Thanks. I know there are issues with the script, but troubleshooting/debugging them was pretty hard with a 45 minute run time. I did get some optimizations in there that sped it up quite a bit. There are reasons the quadtree is storing everything at each level. In c# this isn’t terrible because it’s all pointers, but the js version i have doesn’t keep the duplicates in the tree.

The empty leaf issue is a problem that i still need to fix in my script. if a point falls into an area where the closest node is in a different leaf then it doesn’t find it. I need to put in some expanded bounding boxes for cells that are larger than the leaf bounds.

30 seconds or a minute isn’t so bad since the final import should only really happen once. I’ll take a look at your version. I appreciate the response!

One more question. WangColor doesn’t seem to populate the custom properties in the API, is that expected behavior? Seems like i have to set custom properties at the tileset/terrain level to be able to consume them in js?