use of As New List(Of String)?

just found that The Distinct method is part of System.Linq !! Not a good start

Thanks for example NXJournaling. Much appreciated. Just tested the suggestion on my test function (which takes the data from the input file) and it works with the exception of the last entry not being removed if it's a duplicate!
Below is an extract of the function I am playing/test (GetInputfile() )

Thanks for the help

Dim delim As Char() = {","c}
Dim TotalFile As String
Dim asInputDataSplit(1,3) As String 'array is initialised
Dim astemp(),stemp as String
Dim asDataBlockLine() as string

Dim sr As StreamReader = New StreamReader(openFileDialog1.FileName)

TotalFile=sr.ReadToEnd()
sr.Close

asDataBlockLine = Split(TotalFile, vbLf)

'Resize array asInputDataSplit based on the total number of block
ReDim asInputDataSplit(Ubound(asDataBlockLine),3)

For i = 0 to Ubound(asDataBlockLine)
GetInputfile.Add (Trim(Replace(asDataBlockLine(i),vbLf, "")))
TheLW.WriteLine ("item is " & GetInputfile.Item(i))
Next

Dim uniqueValues As New List(Of String)

For Each temp As String In asDataBlockLine
If Not uniqueValues.Contains(temp) Then
uniqueValues.Add(temp)
End If
Next

'list the unique values
TheLW.WriteLine("unique values:")
For Each temp As String In uniqueValues
TheLW.WriteLine(temp)
Next

That's my guess too but I have to go back to do real work. So will have a look at that later this week. Probably just a check/search for the possible non-printing character as a 1st debugging step

Just spent a few minutes testing the ReadLine() approach rather than reading the whole input text file. Seems to work (see below).
I haven't figured out to get the length of the uniqueValues List so that I can Redim the string array asInputDataSplit() but using a simple counter (k) seems to work. With this approach I can
read each line
check if it is non unique AND valid
if it is add it to the list uniqueValues
I then process the line read (it's unique and valid!)

by doing so I only return an valid array of string to the main function which means that no checks are required in the Main.

Thanks for your help

Regards

JXB
'-----------------------
If openFileDialog1.ShowDialog() = DialogResult.OK Then

TheLW.WriteLine("Input File selected: " & openFileDialog1.FileName)

Using sr As StreamReader = New StreamReader(openFileDialog1.FileName)

Try
sDataBlockLine = sr.ReadLine()

While Not sDataBlockLine Is Nothing
'add the new line to uniqueValues List(of String)
'Resize array asInputDataSplit based on the total number of block

If Not uniqueValues.Contains(sDataBlockLine) And CheckInputValidity(sDataBlockLine) = True Then
uniqueValues.Add(sDataBlockLine)
k +=1
astemp = Split(sDataBlockLine, delim) 'delim is "," Line can be split because it's unique and has no error
ReDim asInputDataSplit(k,3)
'allocate each "cell" to array asInputDataSplit()
asInputDataSplit(k,0) = Trim(astemp(0))
asInputDataSplit(k,1) = Trim(astemp(1))
asInputDataSplit(k,2) = Trim(astemp(2))
asInputDataSplit(k,3) = Trim(astemp(3))
Else
iNotKeptInputLine +=1
End If

sDataBlockLine = sr.ReadLine()
End While

'Close the input file
sr.Close

If iNotKeptInputLine <> 0 Then
TheLW.WriteLine("A total of " & iNotKeptInputLine.ToString & " lines will not be processed as they contained non-valid input")
End if

Catch ex As Exception
MessageBox.Show(ex.Message)
End Try
End Using

Using sr As StreamReader = New StreamReader(openFileDialog1.FileName)

Try
sDataBlockLine = sr.ReadLine()

While Not sDataBlockLine Is Nothing
If Not uniqueValues.Contains(sDataBlockLine) And CheckInputValidity(sDataBlockLine) = True Then
uniqueValues.Add(sDataBlockLine)
Else
iNotKeptInputLine +=1
End If

sDataBlockLine = sr.ReadLine()

End While

sr.Close

Catch ex As Exception
MessageBox.Show(ex.Message)
End Try
End Using

If uniqueValues.Count > 0 Then

ReDim asInputDataSplit(uniqueValues.Count-1,3)

For Each sline as String In uniqueValues
k = uniqueValues.IndexOf (sline)

astemp = Split(sline, delim)

'and allocate each "cell" to array asInputDataSplit()
asInputDataSplit(k,0) = Trim(astemp(0))
asInputDataSplit(k,1) = Trim(astemp(1))
asInputDataSplit(k,2) = Trim(astemp(2))
asInputDataSplit(k,3) = Trim(astemp(3))
Next sline

Return asInputDataSplit