Windows Batch to read file and parse lines into tokens and variables

I've made a good deal of headway by searching this site and learning the ridiculous language that is Windows batch scripting, but I'm now at a point where I'm stuck. I have a text file with a variable number of lines, each of which looks something like:

AA8315,"United States",N777AN,"American Airlines",AAL98,B772,"Boeing 777-223",AAL,"2013-06-11 23:30:47.923","2013-06-12 00:01:14.459"

My batch file:

set THEDATE=2013-06-12
set THEDATABASE=c:\Kinetic\BaseStation\Basestation.sqb
set THECSVFILE=c:\Flights.csv
set THEOUTPUTFILE=c:\FlightsNew.csv
set THISLINE=""

if exist %THECSVFILE% del %THECSVFILE%
if exist %THEOUTPUTFILE% del %THEOUTPUTFILE%

:: allow time for the csv file to be deleted
timeout /t 2 /nobreak

c:\sqlite3.exe -header -csv %THEDATABASE% "select Aircraft.ModeS, Aircraft.ModeSCountry as Country, Aircraft.Registration as Reg, Aircraft.RegisteredOwners as Owner, Flights.Callsign, Aircraft.ICAOTypeCode as Type, Aircraft.Type as Model, Aircraft.OperatorFlagCode as 'Op Flag', Flights.StartTime as 'First Seen', Flights.EndTime as 'Last Seen' from Aircraft INNER JOIN Flights ON (Aircraft.AircraftID=Flights.AircraftID) where Flights.EndTime like '%THEDATE% %%' order by Flights.EndTime DESC;" >> %THECSVFILE%

::allow time for the csv to be written to file
timeout /t 5 /nobreak

::read %THECSVFILE% and loop through each line
for /F "usebackq tokens=* delims=" %%A in (%THECSVFILE%) do (
set the_line=%%A
call :process_line
)

:process_line
for /F "usebackq tokens=1,2,3,4,5,6,7,8,9,10 delims=[,]" %%1 in (%the_line%) do (
set hexcode=%%1
set country=%%2
set reg=%%3
set owner=%%4
set callsign=%%5
set planetype=%%6
set model=%%7
set opflag=%%8
set firstseen=%%9
set lastseen=%%10
set THISLINE=%hexcode%,%country%,%reg%,%owner%,%callsign%,%planetype%,%model%,%opflag%,%firstseen%,%lastseen%
echo %THISLINE% > %THEOUTPUTFILE%
)

(I'm assigning the tokens to variables because I will be doing additional validation and formatting of them later. I need to get this part working first!)

When executed, the script does indeed loop through each line of the file, however it does not seem to be assigning `%%1` to the variable `hexcode`.

The output of the executed command looks like this:

C:\>for /F "usebackq tokens=1,2,3,4,5,6,7,8,9,10 delims=[,]" %1 in (AA8315 "United States" N777AN "American Airlines" AAL98 B772 "Boeing 777-223" AAL "2013-06-11 23:30:47.923" "2013-06-12 00:01:14.459") do (
set hexcode=%1
set country=%2
set reg=%3
set owner=%4
set callsign=%5
set planetype=%6
set model=%7
set opflag=%8
set firstseen=%9
set lastseen=%10
set THISLINE=,"United States" ,N807FD ,"Fedex Express" ,FDX1378 ,,"Airbus A310-324" ,FDX ,"2013-06-12 22:56:54.639" ,"2013-06-12 23:05:31.822"
echo "" 1>c:\FlightsNew.csv
)
The system cannot find the file AA8315.

Any help is greatly appreciated!

You can only use letters fro the metavariable (`%%1` in your code) - but the lower-case and upper-case letters are distinct.

Yes, you can use some other characters, but the contiguous blocks avaliable for `"tokens=1-10"` (which is an easier version of 1,2,3...) are a..z and A..Z

%0..%9 are reserved for the parameters to the batch or batch-procedure.

I have always had problems with comma separated values in a `for` loop. Here's what I did to make your code work.

Test.txt
-
AA8315,"United States",N777AN,"American Airlines",AAL98,B772,"Boeing 777-223",AAL,"2013-06-11 23:30:47.923","2013-06-12 00:01:14.459"

BatchFile.bat
-
set THECSVFILE=test.txt

::read %THECSVFILE% and loop through each line
for /F "usebackq tokens=* delims=" %%A in (%THECSVFILE%) do (
set the_line=%%A
call :process_line
)
goto TheEnd

:process_line
for /F "usebackq tokens=1,2,3,4,5,6,7,8,9,10 delims=~" %%1 in ('%the_line:,=~%') do (
set hexcode=%%1
set country=%%2
set reg=%%3
set owner=%%4
set callsign=%%5
set planetype=%%6
set model=%%7
set opflag=%%8
set firstseen=%%9
set lastseen=%%10
set THISLINE=%hexcode%,%country%,%reg%,%owner%,%callsign%,%planetype%,%model%,%opflag%,%firstseen%,%lastseen%
echo %THISLINE% > %THEOUTPUTFILE%
)

:TheEnd

Notice the **:process_line** `for` loop. I had to add single quotes around the `%the_line%` so it didn't try to interpret the string as a filename. Then I replaced all commas with the ~ character, and used the ~ character as the delimiter. It may not work precisely with all your data (if it contains single quotes or the ~ character), but it does work with this one record and gets you moving in the right direction again.

this works here:
<pre>

for /f "tokens=1-10delims=," %%a in ("AA8315,"United States",N777AN,"American Airlines",AAL98,B772,"Boeing 777-223",AAL,"2013-06-11 23:30:47.923","2013-06-12 00:01:14.459"") do (
set hexcode=%%a
set country=%%b
set reg=%%c
set owner=%%d
set callsegn=%%e
set planefype=%%f
set model=%%g
set opflag=%%h
set firstseen=%%i
set lastseen=%%j
set THISLINE=%%a,%%b,%%c,%%d,%%e,%%f,%%g,%%h,%%i,%%j
)
>"c:\FlightsNew.csv" echo %THISLINE%

</pre>

I'm not sure, why you need the tokens.