When using Visual Studio Code on Ubuntu, despite using the Colemak keyboard layout, the Caps Lock key was not remapped to Backspace in the editor. In order to remap the key manually, I used the following process.

Edit the keybindings.json file by selecting "File" -> "Preferences" -> "Keyboard Shortcuts." In the "Keyboard Shortcuts" window that opens, there's a line which reads "For advanced customizations open and edit keybindings.json", click on "keybindings.json."

Add the following entry to the file:

{"key": "capslock", "command": "deleteLeft"}

After adding this entry, the file contents were as follows:

// Place your key bindings in this file to overwrite the defaults
[
    { "key": "capslock", "command": "deleteLeft"}
]