Downloading Massive Data into AmiBroker
Download 1-minute bars from Massive with the Massive Flat Backfiller and import them into an AmiBroker database, step by step
The Massive Flat Backfiller is a small Windows program you run from PowerShell. It does three jobs, one per run:
- Download 1-minute (or daily) bars from Massive into a folder of CSV files, one file per symbol.
- Make an import file from those CSVs.
- You then import that file into AmiBroker.
The folder of CSVs is not your AmiBroker database. It's a staging area. AmiBroker only sees the data after step 3.
Start with two symbols, SPY and QQQ, to learn the steps. Once they work, run the same steps for the whole market.
Before you start
Pick folders without spaces
You need two folders: one for the program and one for the downloaded CSVs. For example:
M:\MassiveBackfillfor the programM:\MassiveBackfill\barsfor the CSVs
Avoid spaces in folder names. In a command, a space separates one setting from the next, so a path like M:\Ami data Bfill gets read as three separate things. If a path must have spaces, put it in double quotes: --data-dir "M:\Ami data Bfill\bars".
Install the AmiBroker import format (once)
AmiBroker needs to be told how to read the import file. The zip has an AmibrokerFormats folder:
- Copy
mabekit_intraday.formatandmabekit_daily.formatfrom it into AmiBroker'sFormatsfolder, normallyC:\Program Files\AmiBroker\Formats. - Open
C:\Program Files\AmiBroker\Formats\import.typesin Notepad, add these two lines at the end and save:
MabeKit Intraday Import (*.qcolcsv)|*.qcolcsv|mabekit_intraday.format
MabeKit Daily Import (*.qcolcsv)|*.qcolcsv|mabekit_daily.format
Windows may ask for administrator permission to save in C:\Program Files.
Find your Massive keys
In your Massive account, go to Keys > Manage Keys. There are two values, and you need both:
- the Access Key ID
- the Secret Access Key
Treat them like a password: don't paste them into a chat or an email.
Opening PowerShell in the right folder
Every command below assumes PowerShell is open in the program's folder:
- In File Explorer, open the folder with
massiveflatbackfill.exein it. - Right-click an empty spot and choose Open in Terminal. On older Windows versions, Shift + right-click and choose Open PowerShell window here.
- The prompt shows the folder, e.g.
PS M:\MassiveBackfill>.
Check that it works:
.\massiveflatbackfill.exe --help
It should print the program's version and a list of options.
The .\ at the start means "the program in this folder". PowerShell requires it. Without it you'll see an error that the command isn't recognized.
If you typed the .\ and still get the error, PowerShell is open in a different folder from the program:
.\massiveflatbackfill.exe : The term '.\massiveflatbackfill.exe' is not recognized as the name of a cmdlet, function, script file, or operable program.
- Look at the prompt. It shows the folder PowerShell is in, like
PS M:\ami\1mbars>. If that's your CSV folder or your AmiBroker folder, not the one withmassiveflatbackfill.exein it, open PowerShell in the program's folder as above. - List what's in the folder: type
dir *.exeand press Enter. Ifmassiveflatbackfill.exeisn't listed, it's not in this folder. If it's listed under a different name, likemassiveflatbackfill (1).exeafter a second download, rename it, or type the name as listed. - Still in the zip? Opening the zip in File Explorer shows the program, but it can't run from there. Right-click the zip, choose Extract All, and open PowerShell in the folder you extracted to.
You can also skip the folder and type the program's full path instead of .\massiveflatbackfill.exe, like M:\MassiveBackfill\massiveflatbackfill.exe --help.
Reading the examples
Examples show the parts you replace in one of two ways:
- as a placeholder in angle brackets, like
--access-key <id>, - or in capitals, like
PASTE_SECRET_KEY_HERE.
Replace the whole placeholder, brackets included, with your value:
Wrong: --access-key <id> ec7f37de-...
Right: --access-key ec7f37de-...
Typing the < and > makes PowerShell stop with errors like "The '<' operator is reserved for future use" or "Missing expression after unary operator '--'".
Each command is one line. Paste it whole and press Enter.
Step 1: Download SPY and QQQ
.\massiveflatbackfill.exe --access-key YOUR_ACCESS_KEY_ID --secret-key YOUR_SECRET_ACCESS_KEY --symbols SPY,QQQ --barsize 1 --oldest-date 20250101 --data-dir M:\MassiveBackfill\bars
Replace:
YOUR_ACCESS_KEY_IDandYOUR_SECRET_ACCESS_KEYwith your two keys,20250101with the first day you want (YYYYMMDD). Without it you get one year,- the folder after
--data-dirwith your CSV folder.
It prints a line per trading day:
[2025-01-02] 1904 bars across 2 symbol(s) | ...
The end of each line estimates the time left. When it finishes, the folder has SPY_1.csv and QQQ_1.csv.
Save your keys so you don't type them every time (optional)
Run these two lines once, with your keys, then close PowerShell and open a new window:
setx MASSIVE_ACCESS_KEY_ID "YOUR_ACCESS_KEY_ID"
setx MASSIVE_SECRET_ACCESS_KEY "YOUR_SECRET_ACCESS_KEY"
After that, leave --access-key and --secret-key out of the download commands.
Step 2: Make the import file
Run the program again with --import. This pass doesn't download anything and doesn't need your keys:
.\massiveflatbackfill.exe --import --barsize 1 --data-dir M:\MassiveBackfill\bars --import-file M:\MassiveBackfill\all.qcolcsv
Use the same --data-dir as in step 1. It writes all.qcolcsv, every bar from the CSVs in one file.
Step 3: Import into AmiBroker
Create a 1-minute database (once)
- In AmiBroker, choose File > New > Database.
- Pick a folder for the database, e.g.
M:\AmiData\massive01. This is a different folder from the CSVs. - Click Create. In the settings that follow, set Base time interval to 1 minute.
- Leave the data source as (local database), not a live data plugin.
- Click OK.
The base time interval matters. A database created as End-of-day (EOD) stores the bars as daily ones: charts show only daily bars and the intraday intervals are greyed out. Changing the setting later doesn't bring back the minutes already imported, so if that happens, create a new database with 1 minute and import again.
File > Database Settings shows the current database's base interval.
Import the file
- With the 1-minute database open, choose File > Import ASCII...
- In the Files of type dropdown at the bottom, choose MabeKit Intraday Import.
- Open
M:\MassiveBackfill\all.qcolcsv.
The new symbols appear in AmiBroker's symbol list. Open a SPY chart and switch it to 1 minute to check the bars.
If MabeKit Intraday Import isn't in the dropdown, the format setup above hasn't been done. If AmiBroker was open while you did it, restart AmiBroker.
The whole market
Use the same download command without --symbols:
.\massiveflatbackfill.exe --access-key YOUR_ACCESS_KEY_ID --secret-key YOUR_SECRET_ACCESS_KEY --barsize 1 --oldest-date 20250101 --data-dir M:\MassiveBackfill\bars
Then, before step 2, delete the old import file (M:\MassiveBackfill\all.qcolcsv), and run steps 2 and 3 again.
Why: the --import pass only writes bars newer than the last bar in the existing import file, and it uses that one date for every symbol. After the SPY and QQQ test, that date is the end of your test. Without deleting the file, every new symbol gets only the bars after it, often just a day, and charts in AmiBroker won't scroll back further. If that has already happened, delete the file, make it again and import it again. Bars already in the database are replaced, not doubled.
- How long it takes: Massive publishes one file per trading day with every symbol in it, and the program downloads one file per day. So the whole market downloads the same files as two symbols; writing thousands of CSVs instead of two adds time on top. The progress line shows the estimated time left.
- The symbol count on each line is the number of symbols that got new bars that day. It varies from day to day. To see how many symbols you have in total, count the
_1.csvfiles in the CSV folder. - Leave PowerShell open until it prints
=== Backfill completeor=== Backfill finished. - Don't click inside the PowerShell window while it runs, and keep the PC from going to sleep. A click can start selecting text, and Windows pauses the program until the selection ends: if the window's title starts with Select, press Esc and the download carries on.
Keeping it up to date
You don't need a new database or a new folder for each download:
- Download: run the same download command. It starts where the last run stopped and adds only the new days.
- Make the import file: run the same
--importcommand with the same--import-file. It writes only the bars added since the last import file. That's right when you're adding new days to the same symbols. When you add symbols, or download earlier dates, delete the old import file first, as above. - Import: import that file into the same AmiBroker database.
Massive publishes each day's file after the close, so the latest day may not be there until the next morning.
If a download was interrupted
Before you import, check how the download ended. If you watched it finish, the last lines in PowerShell tell you. If PowerShell was closed or your PC restarted during a download, look in the log.
The program writes everything it prints to massiveflatbackfill.log, in the folder you ran it from. Every run adds to the end of it, so open it in Notepad and scroll to the bottom. The last run starts with === Backfill started and the date and time.
- If the last run ends with
=== Backfill complete, every day downloaded. - If it ends with
=== Backfill finished with 2 failed day(s), the run reached the end, but some days didn't download, even after trying each one 4 times. A line a little above it names them, like2 day(s) failed to download: 2026-10-14, 2026-10-15. Fill those days in as below. - If it ends with a day's line, like
[2024-03-15] 1904 bars across 2 symbol(s), the download stopped after that day.
Versions before 1.0.4 don't try a failed day again, and still end with === Backfill complete when a day failed. To see your version, run .\massiveflatbackfill.exe --version. With an older version, search each run in the log for ERROR downloading: each such line names a day that didn't download. The latest version is on the Massive Backfill Tool page.
If it stopped, run the same download command again, with the same --oldest-date. From version 1.0.5 it starts after the last day it finished. Older versions read every day from the oldest date again, so they take about as long as the first run. Either way it only adds bars after the last one in each symbol's CSV. Nothing is doubled. When it prints === Backfill complete, make the import file and import it as usual. If it prints === Backfill finished instead, also fill in the days it lists, as below.
Filling in a day that failed
[21:42:47] [2022-10-07] ERROR downloading: The decryption operation failed, see inner exception.
A day in the failed list, or a line like this in the log, means that day's download broke off, most likely from a network problem, and the program went on to the next day. Running the download again doesn't fill it in: by then every symbol's CSV has bars after that day, and the program only adds bars after the last one.
Download the day into a folder of its own instead, then import it into the same AmiBroker database:
- Download the day.
--oldest-dateis the failed day and--end-dateis the day after it:
.\massiveflatbackfill.exe --barsize 1 --oldest-date 20221007 --end-date 20221008 --data-dir M:\MassiveBackfill\fill
- Make an import file from that folder:
.\massiveflatbackfill.exe --import --barsize 1 --data-dir M:\MassiveBackfill\fill --import-file M:\MassiveBackfill\fill.qcolcsv
- In AmiBroker, with the same database open, choose File > Import ASCII..., pick MabeKit Intraday Import, and open
fill.qcolcsv. The day's bars go into place among the bars already there.
For several failed days in a row, fill them in with one download: --oldest-date is the first failed day and --end-date is the day after the last. For 2026-10-14 through 2026-10-16:
.\massiveflatbackfill.exe --barsize 1 --oldest-date 20261014 --end-date 20261017 --data-dir M:\MassiveBackfill\fill
For failed days that aren't in a row, repeat the steps with a new folder and import file for each, or delete the fill folder and fill.qcolcsv before the next one.
Split adjustments
Massive's files are not adjusted for splits. The program can adjust them with --adjust, but that needs a separate Massive API key. You can skip it while you learn. .\massiveflatbackfill.exe --help describes it.
When a command doesn't work
Copy the command you ran and everything it printed into Claude, ChatGPT or an email to me. The error message almost always says what's wrong:
- a key that's missing or mistyped,
- a folder that doesn't exist,
- a path with spaces and no quotes.
Remove your secret key from the text before you send it.
S3 returned 403 Forbidden
Fatal error: S3 returned 403 Forbidden. Check your access key, secret key, and that your account has flat file access. Endpoint: https://files.massive.com/flatfiles/us_stocks_sip/minute_aggs_v1/2020/01/2020-01-01.csv.gz
Massive refused to send the day at the end of the message. The date in it is the first day the program tried. Holidays and weekends don't cause this: the program skips them.
Run the same command with a recent --oldest-date, like the first day of this year:
- If that downloads, your keys work, but your Massive plan doesn't go back as far as the date you asked for. Use a later
--oldest-date, or upgrade the plan for more history. - If it fails the same way, check both keys in Keys > Manage Keys, and that your Massive plan includes flat files.