You are here

Show times are all zero?

15 posts / 0 new
Last post
bt101
Offline
Joined: 4 months
Last seen: 4 months
Show times are all zero?

Newbie here. Bear with me as I tried to format the post, but it got majorly messed-up, so I had to switch to plain text (I don't have a degree in html/xml).

I managed to get a download of some guide info, however all the show times are zero (I gather midnight).

For example, here is the guide.xml:
=======================================================================
<?xml version="1.0" encoding="UTF-8"?>
<tv generator-info-name="WebGrab+Plus/w MDB &amp; REX Postprocess -- version V5.1.0.0 -- Jan van Straaten" generator-info-url="http://www.webgrabplus.com">
<channel id="CFTO-DV">
<display-name lang="en">CFTO-DV</display-name>
<url>http://www.canada.com</url>
</channel>
<programme start="20240103000000 +0000" stop="20240103000000 +0000" channel="CFTO-DV">
<title lang="en">&lt;\/td&gt;</title>
</programme>
<programme start="20240103000000 +0000" stop="20240103000000 +0000" channel="CFTO-DV">
<title lang="en">Footwear Clearance &lt;\/td&gt;</title>
</programme>
<programme start="20240103000000 +0000" stop="20240103000000 +0000" channel="CFTO-DV">
<title lang="en">Footwear Clearance &lt;\/td&gt;</title>
</programme>
=======================================================================

Using v5.1 on linux Mint 21 (aka Ubuntu 22.04).

Being a newbie, I definitely could have messed-up something.
I was doing well with the linux install page:
http://webgrabplus.com/documentation/installation/linux

However when I got to the configuration page, things went off the rails:
http://webgrabplus.com/documentation/configuration/howto

The page has a lot of info, but after sifting through the chaff, it boils down to making two mods to the xml template config file:
1. Update destination of guide.xml
2. Add a line for each channel

I followed those instructions and after making those two mods, I ended up with this:
=======================================================================
<?xml version="1.0"?>
<settings>
<!-- for detailed info about the settings see http://webgrabplus.com/documentation/configuration/webgrabconfigxml
and http://webgrabplus.com/sites/default/files/downloads/Misc/Documented_Con... -->
<filename>guide.xml</filename>
<mode>
</mode>
<postprocess grab="y" run="n">rex</postprocess>
<user-agent>Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36 Edg/79.0.309.71</user-agent>
<!-- for siteini's that need a decrypt_userkey-->
<decryptkey site="site-name">decrypt_userkey</decryptkey>
<!-- add the correct license id values in the next line -->
<license wg-username="your webgrab+plus username" registered-email="your registered email address" password="your license password">To force a license update; replace this text with the letter f</license>
<logging>on</logging>
<retry time-out="5">4</retry>
<timespan>0</timespan>
<update>f</update>
<!--
Replace the next dummy channel entry with the channels you want.
You can look into the installed siteini.pack folder on your computer

For the latest version,
see http://webgrabplus.com/epg-channels for the available sites/channels
or https://github.com/SilentButeo2/webgrabplus-siteinipack/tree/master/site...
-->
<channel update="i" site="canada.com.L9H1N3" site_id="53" xmltv_id="CFTO-DV">CFTO-DV</channel>
</settings>
=======================================================================

However when I ran it, it just errors:
=======================================================================
user@pc:~/.wg++$ ./run.net.sh

WebGrab+Plus/w MDB & REX Postprocess -- version V5.1.0.0

Jan van Straaten
Francis De Paemeleere

thanks to Paul Weterings, Mat8861, Blackbear199 and all the contributing users
--------------------------------------------------------------------------------

Job started at 02/01/2024 23:33:31
Checking License ..
Program update available !! - Version 3.3.0 @
http://webgrabplus.com/download/sw/v3.3.0
Job finished at 02/01/2024 23:33:38 done in 6s

Index was outside the bounds of the array.

at  .•..ctor(Dictionary`2, Boolean, Boolean, Boolean)
at  . ..ctor(String)
at  .‥. (String[])
Unhandled Exception
For detailed info, see log file /home/user/.wg++/./WGLicense.log.txt
Execution stopped
=======================================================================

So I scrapped using the supplied template file, and just used the one from the instruction web page. I ended up with this:
=======================================================================
<!--?xml version="1.0"?-->
<settings>

<!-- for detailed info about the settings see http://webgrabplus.com/node/30 -->

<filename>guide.xml</filename>
<mode></mode>
<postprocess grab="y" run="n">mdb</postprocess>
<logging>on</logging>
<retry time-out="5">4</retry>
<timespan>6</timespan>
<update>f</update>
<channel update="i" site="canada.com.L9H1N3" site_id="53" xmltv_id="CFTO-DV">CFTO-DV</channel>
</settings>
=======================================================================

That finally worked, but as noted above, I'm getting time=zero for all programs.

bt101
Offline
Joined: 4 months
Last seen: 4 months

Ok, I figured-out one problem. The supplied xml config file is in DOS format (CRLF) and it has spaces in fields that I had to remove, and that is why I was getting an error when trying to use it. That only leaves the problem where my guide.xml file has all zeroes (midnight) for every show time:

Uggh, I can't for the life of me post any of the xml without the post getting messed-up. I've attached the files, including guide.xml which has all zeroes for the times.

Attachments: 
mat8861
Offline
WG++ Team memberDonator
Joined: 8 years
Last seen: 12 hours

The ini you are using is pretty old, if time is all zero, means the correct time is not being scrubbed. Can you try canada.com.T5L4E1 ?

bt101
Offline
Joined: 4 months
Last seen: 4 months
mat8861 wrote:

The ini you are using is pretty old, if time is all zero, means the correct time is not being scrubbed. Can you try canada.com.T5L4E1 ?

Thanks - ok I gave that a try.

It produces an error:

cannot find /home/user/.wg++/canada.com.T5L4E1.cookies.txt !
loadcookie failed! ... cannot update this channel

I searched for that file and cannot find it.

Also, I'm not sure how I would use the T5L4E1 ini file, as it does not contain other stations like CKY, CKND, etc that I found in the L9H1N3 file.

Blackbear199
Offline
Blackbear199's picture
WG++ Team memberDonator
Joined: 8 years
Last seen: 6 days

siteini.pack/Canada folder
read the canada.com.howto.txt

this site will not work without a cookie file.

bt101
Offline
Joined: 4 months
Last seen: 4 months
Blackbear199 wrote:

siteini.pack/Canada folder
read the canada.com.howto.txt
this site will not work without a cookie file.

Thanks.
Ah yes, I see there is a set of instructions in a file called "canada.com.no-provider_info.howtocustomize".
Ok, I'm going though the instructions, and when I get to the part about the cookie, I run into a roadblock.
The instructions say:

B6. Save the cookies in the WG++ home folder as canada.com_cookies.txt
(see online documentation .. how to do that)

When I look at the online instructions, it says to install an extension for Chrome or FF, however those extensions no longer exist. How do I extract the cookie information and put it in a format that webgrab will understand?

Blackbear199
Offline
Blackbear199's picture
WG++ Team memberDonator
Joined: 8 years
Last seen: 6 days

for chrome and edge i use EditThisCookie extension.
in the settings change it so it exports the cookie in netscape format.
thats the format webgrab needs.

heres a updated ini file.
no idea what changed.
for this ini the cookie filename is canada.com.cookies.txt(no underscore used)
if you edit the ini on the site {xxx} line you will see loadcookie=xxxxx.
xxxxx is the filename,use whatever name you want.just name your cookie file to match.

Attachments: 
bt101
Offline
Joined: 4 months
Last seen: 4 months
Blackbear199 wrote:

for chrome and edge i use EditThisCookie extension.
in the settings change it so it exports the cookie in netscape format.
thats the format webgrab needs.
heres a updated ini file.
no idea what changed.
for this ini the cookie filename is canada.com.cookies.txt(no underscore used)
if you edit the ini on the site {xxx} line you will see loadcookie=xxxxx.
xxxxx is the filename,use whatever name you want.just name your cookie file to match.

Thanks for your patience. I hope you won;t regret it :-)
Ok, I've done this a couple of times.

Summary:
-I get some errors when running wg against the cookie (but they *may* not be a problem??)
-After getting past the errors, the channel file only contains one channel, when it should contain 4

When I get the cookie from the browser, it has 5 lines in it.
It is attached as file canada.comR3B1B9_cookies_orig.txt.

When I run wg against this cookie, I get errors that complain about the 2 lines with "data" in them.
For example:

The 'Value'='{"data":{"adLight":false,"adLightDisabled":true,"id":"jdksnrwe77eo03dx57y5mitqbsf0wuni","updated":1704342750598},"exp":604800000,"ts":1704342750598,"mac":-1538500571}' part of the cookie is invalid.

at System.Net.Cookie.VerifySetDefaults(CookieVariant variant, Uri uri, Boolean isLocalDomain, String localDomain, Boolean setDefault, Boolean shouldThrow)
at System.Net.CookieContainer.Add(Cookie cookie)
at System.Net.CookieContainer.Add(CookieCollection cookies)
at  .‬..ctor( , Dictionary`2,  )
at  .‥. (String[])
Unhandled Exception

So I just deleted both of those lines, and ran wg again. It ran through with no error. I attached the resulting cookie file after deleting those 2 lines:
File = canada.comR3B1B9.cookies.txt

Assuming the above is not a concern, then the final problem is that the resulting channel file only has one channel in it (it should have 4). Attached channel file = canada.com.R3B1B9.channels.xml

If I've got it right, it appears the ini file would load this URL:
https://tvmds.tvpassport.com/snippet/white_label/php/grid.php?subid=tvpa...|subpage|&wd=1905

If I put that URL in a browser, it returns a bunch of data, and I can see it includes all 4 channels (CBWT, CKY, CKND, CHMI).
However the resulting channel file only contains the first channel (CBWT).
Interestingly, the channel file has site_id=13 for that CBWT entry, which is incorrect, and happens to correspond to the last item in the returned data (CHMI is 13).

I noticed the returned data has no carriage returns, and everything is essentially on one line. Is the parser that searches for channels, expecting each channel to be on a separate line? The reason a ask, is that the one item that it finds, happens to be the first channel (CBWT), however it assigns site_id=13, which is incorrect, and happens to be the last site_id in the output.
I'm wondering if it thinks there is only one entry because there is only one line, and it grabs the first channel and last site_id.

This is all assuming I didn't make any mistakes (which would be a miracle).

I've also attached these files:
WebGrab++.config.xml
canada.com.R3B1B9.ini
hot_cookies.txt
WebGrab++.log.txt

Attachments: 
Blackbear199
Offline
Blackbear199's picture
WG++ Team memberDonator
Joined: 8 years
Last seen: 6 days

first,good job.
looks like you did everything correctly.
for a new user..well done.

attatched is a fixed ini for your use.
i fixed the channel creation.
i included a channel list i made in the attatchment also.

i also fixed the files in the post above(no provider ini).

you will need to downgrade your webgrab version to V5.0.1
http://webgrabplus.com/download/sw/v5.0.1

this is due to a bug in V5.1.3 with date/time calculations.
if you try the ini with V5.1.3 you will see what i mean,wg will crash with a error about this.

Attachments: 
bt101
Offline
Joined: 4 months
Last seen: 4 months
Blackbear199 wrote:

first,good job.
looks like you did everything correctly.
for a new user..well done.
attatched is a fixed ini for your use.
i fixed the channel creation.
i included a channel list i made in the attatchment also.
i also fixed the files in the post above(no provider ini).
you will need to downgrade your webgrab version to V5.0.1
http://webgrabplus.com/download/sw/v5.0.1
this is due to a bug in V5.1.3 with date/time calculations.
if you try the ini with V5.1.3 you will see what i mean,wg will crash with a error about this.

Thanks - we're so close I can taste it.
I may be doing/seeing something wrong, because my crash experience is the opposite of what you mention.

I inserted the two new files (the channel file and the ini file).
I added the 4 channels to WebGrab++.config.xml.

Just as a sanity check, to see where I stand....I decided to run it without changing any of the dotnet stuff.
It ran completely through with no crash???
Now that I think of it though, the only changes in the new ini file are in the channel grab section, and they are commented out because we already have the channel file. The rest of the ini file is the same as before, and wg did not crash before.
It did, however, produce a guide file, but again with zero times.

I then downloaded the v5.0.1. I see it only contains a bin.net folder. So I got rid of the original bin.net folder and replaced it with the v5.0.1 bin.net folder.

Again, for sanity check, I just ran it.
It ran, with no crash, but did give the error saying that it needs dotnet version 6.

I then installed dotnet version 6.

Now when I run.... it crashes.

The output is:

Failed to create CoreCLR, HRESULT: 0x80070057

I purged the old dotnet7 package, and same crash result.
I went back to look at the download page for v5.0.1 to see if I missed anything (maybe there was special instructions due to the fact that the download only contained a bin.net folder).
It says it needs dotnet6, and the link on the page points to the same instructions for the latest version.

mat8861
Offline
WG++ Team memberDonator
Joined: 8 years
Last seen: 12 hours

Checkout this https://stackoverflow.com/questions/64168113/failed-to-create-coreclr-hr...
I think you have some other version there
On mine i did upgrade in sequence:
root@ubuntu:~# dotnet --info
.NET SDK:
Version: 7.0.404
Commit: 22f221a24c

Ambiente di runtime:
OS Name: ubuntu
OS Version: 20.04
OS Platform: Linux
RID: ubuntu.20.04-x64
Base Path: /usr/share/dotnet/sdk/7.0.404/

Host:
Version: 7.0.14
Architecture: x64
Commit: 808851b07a

.NET SDKs installed:
5.0.408 [/usr/share/dotnet/sdk]
6.0.417 [/usr/share/dotnet/sdk]
7.0.404 [/usr/share/dotnet/sdk]

.NET runtimes installed:
Microsoft.AspNetCore.App 5.0.17 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.25 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.14 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 5.0.17 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.25 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.14 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
None

Environment variables:
Not set

global.json file:
Not found

Learn more:
https://aka.ms/dotnet/info

Download .NET:
https://aka.ms/dotnet/download
root@ubuntu:~#

Blackbear199
Offline
Blackbear199's picture
WG++ Team memberDonator
Joined: 8 years
Last seen: 6 days

what did you install for dotnet 6?
all you need is the .Net Runtime(see screenshot)
u do not need sdk,net desktop runtime or asp.net core runtime.
webgrab is a console app and that all it needs.

bt101
Offline
Joined: 4 months
Last seen: 4 months
Blackbear199 wrote:

what did you install for dotnet 6?
all you need is the .Net Runtime(see screenshot)
u do not need sdk,net desktop runtime or asp.net core runtime.
webgrab is a console app and that all it needs.

Correct, I just installed runtime, and followed the instructions from this page:
http://webgrabplus.com/documentation/installation/linux

Of course that page says to install dotnet v7, so instead I did:
apt install dotnet-runtime-6.0

I did get rid of the old dotnet v7 with an apt purge, but just to be doubly sure, I rolled-back my VM from before the dotnet v7 install. Then did the dotnet v6 install (to be sure there are no remnants of the old v7).

This is what is currently on the machine:

===================================================================================
user@mint:/$ dpkg -l | grep dotnet
ii dotnet-host 6.0.125-0ubuntu1~22.04.1 amd64 dotNET host command line
ii dotnet-hostfxr-6.0 6.0.125-0ubuntu1~22.04.1 amd64 dotNET host resolver
ii dotnet-runtime-6.0 6.0.125-0ubuntu1~22.04.1 amd64 dotNET runtime

===================================================================================
user@mint:/$ apt-cache policy dotnet-runtime-6.0
dotnet-runtime-6.0:
Installed: 6.0.125-0ubuntu1~22.04.1
Candidate: 6.0.125-0ubuntu1~22.04.1
Version table:
*** 6.0.125-0ubuntu1~22.04.1 500
500 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages
500 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages
100 /var/lib/dpkg/status

===================================================================================
user@mint:/$ dotnet --info

global.json file:
Not found

Host:
Version: 6.0.25
Architecture: x64
Commit: 492abbeef0

.NET SDKs installed:
No SDKs were found.

.NET runtimes installed:
Microsoft.NETCore.App 6.0.25 [/usr/lib/dotnet/shared/Microsoft.NETCore.App]

Download .NET:
https://aka.ms/dotnet-download

Learn about .NET Runtimes and SDKs:
https://aka.ms/dotnet/runtimes-sdk-info
===================================================================================

I'm looking at some of the articles about this error, but nothing is clicking with me. As mat8861 mentioned, the error usually seems to be due to having conflicting dotnet versions installed. But I'm pretty sure I've eliminated that possibility.

bt101
Offline
Joined: 4 months
Last seen: 4 months

Ok, as the veterinarian said to professor Schrödinger about his cat "I have some good news, and some bad news".

Good news... I found a post where someone said that dotnet does not like to run on programs that have a symbolic link in the path. So I didn't have a symbolic link in the path, but the path was a file share (I was storing/running everything on a share because I was doing so many tries on different machines). Anyway, I moved everything off the share to a local folder, and it now runs dotnet v6 with no error.

The bad news...
Get ready to laugh.. we've come full circle... the guide.xml still has all zero times.

I've attached the guide.xml and output log.

Maybe I'm cursed ;-)

I assume that (fro post #8 above) if I put the url from the ini file into a browser, and I get data back... that must mean that the internets are sending me the correct channel/programs/times information?

Attachments: 
Blackbear199
Offline
Blackbear199's picture
WG++ Team memberDonator
Joined: 8 years
Last seen: 6 days

i see the problem.
your not a donator.

normally this a ini would work for you as a registered user.
but not canada.com
this is because registered users cannot get show details page.
you get the index page only.
99% of ini have the title and the show start(and stop or duration) on the index page.
canada .com has the show start time on the details page which u cannot get as a registered use.

use tvpassport(international folder).

it uses a multi step to create channel list.
first add this channel line
<channel update="i" site="tvpassport.com" site_id="R3B1B9" xmltv_id="dummy">dummy</channel>
and set
<update>c1-R3B1B9</update>
run webgrab,you will see a tvpassport.com.R3B1B9.channels.xml

copy the channel line you wish to create a channel list for from the file(i assume you want the local ota) and replace the channel line u used above.
<update>c2-R3B1B9-OTA</update>
R3B1B9-OTA can be anything you want,its used to identify the channel list name.
run webgrab and you should have your channel list.

edit the tvpassport.com.ini and change the timezone on the site {xxx} line to the local timezone of pc running webgrab.
America/Winnipeg
or
Canada/Central
are a few valid timezones for winnipeg.
its case sensitive so enter exactly as above.
if webgrab error with msg about timezone,check you log and a list of valid timezones will be there.

you dont do this for every ini you use,just some.
see the remarks near the top of the ini.

Log in or register to post comments

Brought to you by Jan van Straaten

Program Development - Jan van Straaten ------- Web design - Francis De Paemeleere
Supported by: servercare.nl