Create an account

Very important

  • To access the important data of the forums, you must be active in each forum and especially in the leaks and database leaks section, send data and after sending the data and activity, data and important content will be opened and visible for you.
  • You will only see chat messages from people who are at or below your level.
  • More than 500,000 database leaks and millions of account leaks are waiting for you, so access and view with more activity.
  • Many important data are inactive and inaccessible for you, so open them with activity. (This will be done automatically)


Thread Rating:
  • 444 Vote(s) - 3.47 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How can I split a binary file into chunks with certain size with batch script without external software?

#1
There are many reasons to want to split file into chunks - mainly for network transfer (e.g. e-mail attachments) but I'm sure there are scenarios that could require such thing I cannot imagine.
So how can a file be split into chunks that can be easily assembled back to original file (including non-windows systems) ?


What are possibilities:

1. [MAKECAB][1] - the built-in Windows archiver - it can compress file and
split it , but will be difficult to assemble the file on non-Windows
machine.
2. WSH/Jscript/VBscript - binary files can be easy processed with ADODB
Streams.And according to me JScript is preferable.
3. .NET/JScript.net/VB/C# - As .NET comes with command line compilers
it also can be used for this purpose . .NET binary streams probably
are the best in this case that it can offer .
4. [CERTUTIL][2] - as it can convert a binary file to HEX and back it's
possible to process the HEX data with batch file and split the file
to chunks.

Probably GZipStreams can be used also as they allows
reading bytes (and offer portable compression !).I promise an upvote If anyone succeed with this (or
offer any other method) :-)


[1]:

[To see links please register here]

[2]:

[To see links please register here]

Reply

#2
The example below will split a file, producing multiple output files all smaller than the `maxChunkSize` supplied. To reassemble, you can use `copy /b`.

SplitFile.cs
<br/>(compile with `c:\Windows\Microsoft.NET\Framework\v2.0.50727\csc /out:splitFile.exe SplitFile.cs`)

using System;
using System.IO;

namespace SplitFile
{
class Program
{
static void Main(string[] args)
{
long maxChunkSize;
if (args.Length != 3 || !long.TryParse(args[2], out maxChunkSize) || maxChunkSize <= 81920)
{
Console.WriteLine("Usage: splitfile.exe inputFile outputprefix maxchunksize");
Console.WriteLine(" inputfile: File to split");
Console.WriteLine(" outputprefix: Prefix to use for the output name");
Console.WriteLine(" Ex: out -> { out0001.bin, out0002.bin }");
Console.WriteLine(" maxchunksize: Maximum number of bytes in each file");
Console.WriteLine(" Note: this is the maximum size, not an exact size");
Console.WriteLine(" Note: chunk size cannot be smaller than 81920 bytes");
return;
}

string inputFilePath = args[0];
string outputFilePathFormat = string.Format("{0}{{0:0000}}.bin", args[1]);

using (Stream fsInput = File.Open(inputFilePath, FileMode.Open, FileAccess.Read, FileShare.Read))
{
byte[] buffer = new byte[81920 /* default from System.Stream */];
int cOutFileNo = 0;
Stream destination = getOutputFile(ref cOutFileNo, outputFilePathFormat);
try
{
int read;
while ((read = fsInput.Read(buffer, 0, buffer.Length)) != 0)
{
if (destination.Length + read > maxChunkSize)
{
destination.Dispose();
destination = getOutputFile(ref cOutFileNo, outputFilePathFormat);
}

destination.Write(buffer, 0, read);
}
}
finally
{
destination.Dispose();
}
}
}

private static Stream getOutputFile(ref int cOutFileNo, string outFileFormat)
{
string filename = string.Format(outFileFormat, cOutFileNo);
cOutFileNo++;

return File.Open(filename, FileMode.CreateNew, FileAccess.Write, FileShare.None);
}
}
}

Example Use:

<pre>
C:\drop>splitFile.exe ubuntu-rescue-remix-12-04.iso Ubuntu_Split_ 10485760
C:\drop>dir
01/29/2015 17:21 244,570,112 ubuntu-rescue-remix-12-04.iso
01/30/2015 15:27 10,485,760 Ubuntu_Split_0000.bin
01/30/2015 15:27 10,485,760 Ubuntu_Split_0001.bin
01/30/2015 15:27 10,485,760 Ubuntu_Split_0002.bin
01/30/2015 15:27 10,485,760 Ubuntu_Split_0003.bin
01/30/2015 15:27 10,485,760 Ubuntu_Split_0004.bin
01/30/2015 15:27 10,485,760 Ubuntu_Split_0005.bin
01/30/2015 15:27 10,485,760 Ubuntu_Split_0006.bin
01/30/2015 15:27 10,485,760 Ubuntu_Split_0007.bin
01/30/2015 15:27 10,485,760 Ubuntu_Split_0008.bin
01/30/2015 15:27 10,485,760 Ubuntu_Split_0009.bin
01/30/2015 15:27 10,485,760 Ubuntu_Split_0010.bin
01/30/2015 15:27 10,485,760 Ubuntu_Split_0011.bin
01/30/2015 15:27 10,485,760 Ubuntu_Split_0012.bin
01/30/2015 15:27 10,485,760 Ubuntu_Split_0013.bin
01/30/2015 15:27 10,485,760 Ubuntu_Split_0014.bin
01/30/2015 15:27 10,485,760 Ubuntu_Split_0015.bin
01/30/2015 15:27 10,485,760 Ubuntu_Split_0016.bin
01/30/2015 15:27 10,485,760 Ubuntu_Split_0017.bin
01/30/2015 15:27 10,485,760 Ubuntu_Split_0018.bin
01/30/2015 15:27 10,485,760 Ubuntu_Split_0019.bin
01/30/2015 15:27 10,485,760 Ubuntu_Split_0020.bin
01/30/2015 15:27 10,485,760 Ubuntu_Split_0021.bin
01/30/2015 15:27 10,485,760 Ubuntu_Split_0022.bin
01/30/2015 15:27 3,397,632 Ubuntu_Split_0023.bin

C:\drop>copy /b Ubuntu_Split_* Ubuntu_recombined.iso
Ubuntu_Split_0000.bin
Ubuntu_Split_0001.bin
Ubuntu_Split_0002.bin
Ubuntu_Split_0003.bin
Ubuntu_Split_0004.bin
Ubuntu_Split_0005.bin
Ubuntu_Split_0006.bin
Ubuntu_Split_0007.bin
Ubuntu_Split_0008.bin
Ubuntu_Split_0009.bin
Ubuntu_Split_0010.bin
Ubuntu_Split_0011.bin
Ubuntu_Split_0012.bin
Ubuntu_Split_0013.bin
Ubuntu_Split_0014.bin
Ubuntu_Split_0015.bin
Ubuntu_Split_0016.bin
Ubuntu_Split_0017.bin
Ubuntu_Split_0018.bin
Ubuntu_Split_0019.bin
Ubuntu_Split_0020.bin
Ubuntu_Split_0021.bin
Ubuntu_Split_0022.bin
Ubuntu_Split_0023.bin
1 file(s) copied.

C:\drop>dir Ubuntu*.iso
01/29/2015 17:21 244,570,112 ubuntu-rescue-remix-12-04.iso
01/30/2015 15:27 244,570,112 Ubuntu_recombined.iso

C:\drop>fciv -sha1 ubuntu-rescue-remix-12-04.iso
//
// File Checksum Integrity Verifier version 2.05.
//
02403c37cbdb3e03e00f5176807a793ef63d877c ubuntu-rescue-remix-12-04.iso

C:\drop>fciv -sha1 Ubuntu_recombined.iso
//
// File Checksum Integrity Verifier version 2.05.
//
02403c37cbdb3e03e00f5176807a793ef63d877c ubuntu-rescue-remix-12-04.iso

C:\drop>
</pre>
Reply

#3
With all scripts syntax is the same - the file to split and the size in bytes.

1) **MAKECAB** - the main limitation is the usage on Unix/Mac machines.For unix eventually [cabextract][1] or 7zip could be used , but I'm not sure if it can handle split CAB file .Even on windows the EXPAND command cant do it
and EXTRAC32 (command is given in the help message) should be used (or [Shell.Application][2] )

;@echo off

;;;;; rem start of the batch part ;;;;;
;;
;;
;; rem the first parameter is the file you want to split the second is the size in bytes.
;; rem size is not guaranteed but will be not overflown
;;
; if "%~2" equ "" (
; call :helpmessage
; exit /b 1
;)
;if not exist "%~dpnx1" (
; call :helpmessage
; exit /b 2
;)
;if exist "%~dpnx1\" (
; call :helpmessage
; exit /b 3
;)
; rem remove leading zeroes
; cmd /c exit /b %~2
; set /a size=%errorlevel%
; if %size% equ 0 (
; echo size must be greater than 0
; exit /b 4
;)
; rem MaxDiskSize must be multiple of 512 and closest possible to desired size.
;if %~2 LSS 512 set diskSize=512 else (
; set /a part=%~2%%512
; set /a diskSize=%~2-part
;)
;makecab /d the_file="%~1" /d diskSize=%diskSize% /d the_size="%~2" /F "%~dpfnxs0"
;exit /b %errorlevel%
;:helpmessage
; echo no existing file has been passed
; echo usage [split a file to cab parts with given size]:
; echo %~nx0 file size
; echo(
; echo size must be greater than 0
; echo (
; echo for extraction use :
; echo extrac32 /a /e file.ext_part1.cab /l .
; exit /b 0

;;
;;;; rem end of the batch part ;;;;;

;;;; directives part ;;;;;
;;
.New Cabinet
.set GenerateInf=OFF
.Set Cabinet=on
.Set Compress=on
.Set MaxDiskSize=%diskSize%;
.Set MaxCabinetSize=%the_size%
.set CabinetFileCountThreshold=1
.set CompressedFileExtensionChar=_
.Set CabinetNameTemplate=%the_file%_part*.cab
.set DestinationDir=.
.Set DiskDirectoryTemplate=;

.set RptFileName=nul
.set UniqueFiles=ON
;.set DoNotCopyFiles=ON
;.set MaxDiskFileCount=1
.set MaxErrors=1
.set GenerateInf=OFF
%the_file% /inf=no
;;
;;;; end of directives part ;;;;;


--*All other methods are direct splitting and files can be assembled with appending them to each other in correct order* --


2) **JScript** - a hybrid file that must be saved with .bat extension

@if (@x)==(@y) @end /***** jscript comment ******
@echo off
cscript //E:JScript //nologo "%~f0" "%~nx0" %*
exit /b %errorlevel%

@if (@x)==(@y) @end ****** end comment *********/

//https://github.com/npocmaka/batch.scripts/blob/master/hybrids/jscript/zipjs.bat

var FileSystemObj = new ActiveXObject("Scripting.FileSystemObject");
var AdoDBObj = new ActiveXObject("ADODB.Stream");

var ARGS = WScript.Arguments;
var scriptName=ARGS.Item(0);

if (ARGS.length <3) {
WScript.Echo("Wrong arguments");
WScript.Echo("usage:");
WScript.Echo(scriptName +"file_to_split size_in_bytes");
WScript.Quit(1);
}
var file=ARGS.Item(1);
var max_size=parseInt(ARGS.Item(2));

function getSize(file){
return FileSystemObj.getFile(file).size;
}

function isExist(file){
return FileSystemObj.FileExists(file);
}

function writeFile(fileName,data ){
AdoDBObj.Type = 1;
AdoDBObj.Open();
AdoDBObj.Position=0;
AdoDBObj.Write(data);
AdoDBObj.SaveToFile(fileName,2);
AdoDBObj.Close();
}

function readFile(fileName,size,position){
AdoDBObj.Type = 1;
AdoDBObj.Open();
AdoDBObj.LoadFromFile(fileName);
AdoDBObj.Position=position;
fileBytes=AdoDBObj.Read(size);
AdoDBObj.Close();
return fileBytes;


}

function chunker(file,size){
var part=0;
var position=0;
var buffer=readFile(file,size,0);
file_size=getSize(file);
while (buffer !== null ) {
part++;
WScript.Echo("Creating: "+file+".part."+part);
writeFile(file+".part."+part,buffer);
if (size*part <= file_size) {
position=size*part;
} else {
position=file_size;
}
buffer=readFile(file,size,position);
}
}

if (!isExist(file)){
WScript.Echo(file+" does not exist");
WScript.Quit(2);
}

if(max_size<=0){
WScript.Echo("Size must be bigger than 0.")
WScript.Quit(3);
}

chunker(file,max_size);

3) **JScript.net** - self compiled hybrid that must be saved with .bat extension.

@if (@X)==(@Y) @end /* JScript comment
@echo off
setlocal

for /f "tokens=* delims=" %%v in ('dir /b /s /a:-d /o:-n "%SystemRoot%\Microsoft.NET\Framework\*jsc.exe"') do (
set "jsc=%%v"
)

::if not exist "%~n0.exe" (
"%jsc%" /nologo /out:"%~n0.exe" "%~dpsfnx0"
::)

%~n0.exe %*

endlocal & exit /b %errorlevel%

*/

import System;
import System.IO;

var arguments:String[] = Environment.GetCommandLineArgs();

if (arguments.length<3){
Console.WriteLine("Wrong arguments");
Console.WriteLine("usage:");
Console.WriteLine(arguments[0]+"file_to_split size_in_bytes");
Environment.Exit(1);
}

var file=arguments[1];
var max_size=parseInt(arguments[2]);

if (max_size<=0){
Console.WriteLine("size must be bigger than zero");
Environment.Exit(2);
}

if (!File.Exists(file)){
Console.WriteLine("file"+file+" does not exist");
Environment.Exit(3);
}

function writeData(file,data:byte[]){
Console.WriteLine(data.Length);
var writer = new BinaryWriter(File.Open(file, FileMode.Create));
writer.Write(data);
writer.Close();
}

function chunker(inputFile, chunkSize){

var part=0;
var reader= new BinaryReader(File.Open(inputFile, FileMode.Open));
var data:byte[]=reader.ReadBytes(chunkSize);

while(reader.BaseStream.Position !== reader.BaseStream.Length) {
part++;
Console.WriteLine("Processing part " + part);
writeData(inputFile+".part."+part,data);
data=reader.ReadBytes(chunkSize);

}
if (data.Length !== 0) {
part++;
Console.WriteLine("Processing part " + part)
writeData(inputFile+".part."+part,data);
}
reader.Close();
}

chunker(file,max_size);


4) **CERTUTIL** - rather an experimental stuff - it's slower as the buffers are limited to the max length that can string have 8,1** characters and is rather text processing:

@echo off

setlocal enableExtensions
rem :-----------------------------------------
rem : check if should prompt the help message
rem :-----------------------------------------
if "%~2" equ "" goto :help
for %%H in (/h -h /help -help) do (
if /I "%~1" equ "%%H" goto :help
)
if not exist "%~1" echo file does not exist & exit /b 1


rem :-----------------------------------------
rem : validate input
rem :-----------------------------------------
set /a size=%~2
if not defined size echo something wrong with size parameter & exit /b 2
if %size%0 LSS 00 echo not a valid number passed as a parameter & exit /b 3

rem : -- two hex symbols and an empty space are 1 byte
rem : -- so the sum of all hex symbols
rem : -- per part should be doubled
set /a len=%size%*2
set "file=%~dfn1"

for %%F in ("%file%") do set fn=%%~nxF

rem : -- clear temp data
del /F /Q "%temp%\file" >nul 2>&1
del /F /Q "%temp%\fn.p.*" >nul 2>&1
certutil -encodehex -f "%file%" "%temp%\file" >nul
set "part=1"

setlocal enableDelayedExpansion
set "hex_str="
set hex_len=0
break>%temp%\fn.p.!part!

rem : -- reads the hex encoded file
rem : -- and make it on a parts that will
rem : -- decoded with certutil

rem :-- the delimitier is <tab> wich separates
rem :-- line number from the rest of the information
rem :-- in the hex file
rem :---------------------------- v <tab>
for /f "usebackq tokens=2 delims= " %%A in ("%temp%\file") do (
set "line=%%A"
rem : -- there's a double space in the middle of the line
rem :-- so here the line is get
set hex_str=!hex_str!!line:~0,48!
rem echo hex_str !hex_str!
rem :-- empty spaces are cleared
set hex_str=!hex_str: =!
rem echo hex_str !hex_str!
rem :-- the length of the hex line 32
set /a hex_len=hex_len+32

rem : -- len/size is reached
rem : -- and the content is printed to a hex file
if !hex_len! GEQ !len! (
echo !hex_len! GEQ !len!
set /a rest=hex_len-len
for %%A in (!rest!) do (
(echo(!hex_str:~0,-%%A!)>>%temp%\fn.p.!part!
rem : -- the rest of the content of the line is saved
set hex_str=!hex_str:~-%%A!
set /a hex_len=rest
echo !hex_len!
)
certutil -decodehex -f %temp%\fn.p.!part! %fn%.part.!part! >nul
echo -- !part!th part created --
rem :-- preprarin next hex file
set /a part=part+1
break>%temp%\fn.p.!part!
rem :-- reinitilization of the len/size of the file part
set /a len=%size%*2
)
rem : -- a buffer that not allows to
rem : -- to enter too long commmands
rem : -- used to reduce disk operations
if !hex_len! GEQ 7800 (
(echo(!hex_str!)>>%temp%\fn.p.!part!
set "hex_str="
set hex_len=0
rem :-- the size that need to be reached is reduces
rem :-- as there's alredy part of the part of the file
rem :-- added to the hex file
set /a len=!len!-7800
if !len! LSS 0 set len=0

)

)
rem : -- adding the rest of the file
echo !hex_str!>>%temp%\fn.p.!part!
certutil -decodehex -f %temp%\fn.p.!part! %fn%.part.!part! >nul
echo -- !part!th part created --

rem : -- clear created temp data
rem del /F /Q %temp%\fn.p.* >nul 2>&1
rem del /F /Q %temp%\file >nul 2>&1
endlocal
endlocal

goto :eof
rem :-----------------------------------------
rem : help message
rem :-----------------------------------------

:help
echo\
echo Splits a file on parts by given size in bytes in 'pure' batch.
echo\
echo\
echo %0 file size
echo\
echo\


here's a script that can assemble files split with the last three methods:

@echo off
if "%~1" EQU "" echo parameter not entered & exit /b 1
set "parts=%~1.part"

setlocal enableDelayedExpansion
set numb=0
for /f "delims=." %%P in ('dir /b %parts%*') do (
set /a numb=numb+1
)
rem echo !numb!


setlocal enableDelayedExpansion
set "string=%~1.part.1"
for /l %%n in (2;1;!numb!) do (
set "string=!string!+!parts!.%%n"
)
rem echo !string!
copy /y /b !string! %~1%~x1
endlocal


[1]:

[To see links please register here]

[2]:

[To see links please register here]

Reply

#4
Some time ago I wrote a Batch-JScript hybrid script called BinToBat.bat with this purpose. This is its help screen:

Create an installer Batch program for data files of any type

BINTOBAT [/T:.ext1.ext2...] [/L:lineSize] [/F[:fileSize]] filename ...

/T:.ext1.ext2 Specify the extensions of text type files that will not be
encoded as hexadecimal digits, but preserved as text.
/L:lineSize Specify the size of output lines (default: 78).
/F[:fileSize] /F switch specify to generate a Full installer file.
The optional fileSize specify the maximum output file size.

BinToBat encode the given data files as hexadecimal digits (or preserve they
as compressed text) and insert they into InstallFiles.bat program; when this
program run, it generates the original data files.

You may rename the InstallFiles.bat program as you wish, but preserving the
"Install" prefix is suggested.

You may use wild-cards in the filename list.

If the /F switch is not given, a Partial installer is created:
- You may insert a short description for each file.
- You may insert divisions in the file listing via a dash in the parameters.
- The installer allows to select which files will be downloaded and ask
before overwrite existent files.

If the /F switch is given, a Full installer is created:
- The installer always download all files.
- You may specify commands that will be executed after the files were copied.
- You may specify the maximum size of the output file via /F:fileFize, so in
this case the output file will be divided in parts with a numeric postfix.

If you use /F switch you can NOT rename the InstallFiles??.bat files; the
first one is the installer and the rest just contain data.


You may download BinToBat.bat program from [this site][1].


[1]:

[To see links please register here]


Reply

#5
For raw byte chunks, powershell can split files:

set size=<split size>
set file=<file.in>

for %j in (%file%) do (
set /a chunks=%~zj/%size% >nul

for /l %i in (0,1,!chunks!) do (
set /a tail=%~zj-%i*%size% >nul
powershell gc %file% -Encoding byte -Tail !tail! ^| sc %file%_%i -Encoding byte
if %i lss !chunks! FSUTIL file seteof %file%_%i %size% >nul
)
)

These can then be joined by `copy`

Some systems might need to be prepared to run FSUTIL & enable delayed expansion.

There's **another method** which extracts smaller parts of a file but the `makecab` utility encodes the chunks in it's own format so they can't be treated as just raw bytes, similar to a flat binary file eg. via `copy`.

Unlike what some have stated in some answers, the size of the split files **need NOT be a multiple** of 512. The chunks, however can then be joined by `extrac32`.

First make a ddf (text) file or just provide options via `/d` on command line:

.Set CabinetNameTemplate=test_*.cab; <-- Enter chunk name format
.Set MaxDiskSize=900000; <-- Enter file split/chunk size (doesn't have to be multiple of 512)
.Set ClusterSize=1000
.Set Cabinet=on;
.Set Compress=off; <-- Optional: set compression on to save disk space
.set CompressionType=LZX;
.set CompressionMemory=21
.Set DiskDirectoryTemplate=;
file.in <-- Enter the file name to be split

Then:

makecab /f ddf.txt

To get the original file back, ensure all chunks are in the same directory, then call 1st file in sequence:

extrac32 test_1.cab file.out

MakeCAB introduces the concept of a folder to refer to a contiguous set of compressed bytes. <br>
"MakeCAB takes all of the files in the product or application being compressed, lays the bytes down as one continuous byte stream, compresses the entire stream, chopping it up into folders as appropriate, and then fills up one or more cabinets with the folders."

A **third method** to split files via CMD & certutil:

set file="x.7z" &REM compressed to generate CRLF pairs
set max=70000000 &REM certutil has max file limit around 74MB

REM Findstr line limit 8k
REM Workaround: wrap in 7z archive to generate CRLF pairs

for %i in (%file%) do (
set /a num=%~zi/%max% >nul &REM No. of chunks
set /a last=%~zi%%max% >nul &REM size of last chunk
if %last%==0 set /a num=num-1 &REM ove zero byte chunk
set size=%~zi
)

ren %file% %file%.0

for /l %i in (1 1 %num%) do (
set /a s1=%i*%max% >nul
set /a s2="(%i+1)*%max%" >nul
set /a prev=%i-1 >nul

echo Writing %file%.%i
type %file%.!prev! | (
(for /l %j in (1 1 %max%) do pause)>nul& findstr "^"> %file%.%i)

FSUTIL file seteof %file%.!prev! %max% >nul
)
if not %last%==0 FSUTIL file seteof %file%.%num% %last% >nul
echo Done.

**Notes**
1. Chunks can be joined by `copy /b`
2. Filename extensions can be made neater by padding chunk numbers
3. Can be looped to split entire directories

See example output below:

29/04/2022 22:14 71,000,000 ATOMIC Blonde.mp4.014
29/04/2022 23:33 71,000,000 ATOMIC Blonde.mp4.015
30/04/2022 00:56 71,000,000 ATOMIC Blonde.mp4.016
30/04/2022 02:16 71,000,000 ATOMIC Blonde.mp4.017
30/04/2022 02:16 71,000,000 ATOMIC Blonde.mp4.018
30/04/2022 03:05 37,601,635 The Young Messiah.mp4.000
30/04/2022 04:25 71,000,000 The Young Messiah.mp4.001
30/04/2022 05:45 71,000,000 The Young Messiah.mp4.002
30/04/2022 07:05 71,000,000 The Young Messiah.mp4.003
30/04/2022 08:25 71,000,000 The Young Messiah.mp4.004
30/04/2022 09:59 71,000,000 The Young Messiah.mp4.005
30/04/2022 11:23 71,000,000 The Young Messiah.mp4.006
30/04/2022 12:47 71,000,000 The Young Messiah.mp4.007


Tested on Win 10.




Reply



Forum Jump:


Users browsing this thread:
1 Guest(s)

©0Day  2016 - 2023 | All Rights Reserved.  Made with    for the community. Connected through