In-core processing holds all the data in the memory.

Select[Range[100],PrimeQ[#]&]

Map[#^2&,Range[10]]

Out-of-core processing leaves data on the disk, then processes them one piece at a time.

Write the data to the disk

data=Range[10]

fileName=FileNameJoin[{NotebookDirectory[],"testfile.txt"}];

fileWriteHandle=OpenWrite[fileName]

WriteLine[fileWriteHandle,"there"]

Write[fileWriteHandle,"here"]

Map[WriteLine[fileWriteHandle,#]&,data];

Close[fileWriteHandle]

Now do the processing reading one line at a time.  This way, the computer is holding only one piece of data at a time.

sourceFileName=FileNameJoin[{NotebookDirectory[],"testfile.txt"}];

fileReadHandle=OpenRead[sourceFileName]

Read[fileReadHandle,Expression]

Close[sourceFileName]

Automate the process by using a While loop in place of Map

sourceFileName=FileNameJoin[{NotebookDirectory[],"inputFile.txt"}];
targetFileName=FileNameJoin[{NotebookDirectory[],"outputFile.txt"}];

fileReadHandle=OpenRead[sourceFileName];
fileWriteHandle=OpenWrite[targetFileName];

value=1;
While[(value=Read[fileReadHandle,Expression])=!=EndOfFile,
WriteLine[fileWriteHandle,(value^2)];
]

Close[sourceFileName];
Close[fileWriteHandle];

Let While substitute for Select

sourceFileName=FileNameJoin[{NotebookDirectory[],"inputFile.txt"}];
targetFileName=FileNameJoin[{NotebookDirectory[],"outputFile.txt"}];

fileReadHandle=OpenRead[sourceFileName];
fileWriteHandle=OpenWrite[targetFileName];

While[(value=Read[fileReadHandle,Expression])=!=EndOfFile,
If[PrimeQ[value],WriteLine[fileWriteHandle,value]];
]

Close[sourceFileName];
Close[fileWriteHandle];

Let While substitute for Select when it crashes often

sourceFileName=FileNameJoin[{NotebookDirectory[],"inputFile.txt"}];
targetFileName=FileNameJoin[{NotebookDirectory[],"outputFile.txt"}];

Export[sourceFileName,ToString[#]&/@Range[10],"Text"]

Import[sourceFileName]

Import[targetFileName]

fileReadHandle=OpenRead[sourceFileName];

fileWriteHandle=OpenWrite[targetFileName];
Close[fileWriteHandle];

While[(value=Read[fileReadHandle,Expression])=!=EndOfFile,
If[PrimeQ[value],
fileWriteHandle=OpenAppend[targetFileName];
WriteLine[fileWriteHandle,value];
Close[fileWriteHandle];
];
]

While[(value=Read[fileReadHandle,Expression])=!=EndOfFile,
fileWriteHandle=OpenAppend[targetFileName];
WriteLine[fileWriteHandle,{value,PrimeQ[value]}];
Close[fileWriteHandle];
]

Close[sourceFileName];
Close[fileWriteHandle];

How to handle more complicated data types such as entities?

regions=(Entity["Philippines", Entity["Country","Philippines"], "Entity[\"Country\", \"Philippines\"]", "country"])[(EntityProperty["regions", EntityProperty["Country","AdministrativeDivisions"], "EntityProperty[\"Country\", \"AdministrativeDivisions\"]"])]

entityFileName=FileNameJoin[{NotebookDirectory[],"entityfile.txt"}];

fileWriteHandle=OpenWrite[entityFileName,FormatType->InputForm];

Map[Write[fileWriteHandle,#]&,regions];

Close[entityFileName];

Read the entities

entityFileName=FileNameJoin[{NotebookDirectory[],"entityfile.txt"}];

fileReadHandle=OpenRead[entityFileName]

Read[fileReadHandle,Expression]

Close[entityFileName]