--- title: "URLFetchAsynchronous" language: "en" type: "Symbol" summary: "As of Version 11, URLFetchAsynchronous has been superseded by URLSubmit." keywords: - wget - background web canonical_url: "https://reference.wolfram.com/language/ref/URLFetchAsynchronous.html" source: "Wolfram Language Documentation" --- # URLFetchAsynchronous ⚠ As of Version 11, ``URLFetchAsynchronous`` has been superseded by ``URLSubmit``. URLFetchAsynchronous[url, func] performs a connection in the background, calling func when an event is raised. ## Details and Options * Types of events that may be raised: | | | | ------------ | ---------------------------------------------------------------------------------------------- | | "data" | returns a list of bytes and indicates that the connection has completed | | "progress" | returns information about the current connection of the form {dlnow, dltotal, ulnow, ultotal} | | "error" | indicates that an error occurred while attempting to connect to a URL | | "headers" | headers received from the web server | | "cookies" | a list of all known cookies | | "statuscode" | the status code returned by the server | * The following options can be given: | | | | | --- | --- | --- | | Method | "GET" | method to use for request | | "Parameters" | {} | parameters to be sent for the request | | "Body" | "" | contents of message body to be sent | | "MultipartElements" | {} | list of multipart data elements to send | | "Username" | "" | username to use for the request | | "Password" | "" | password to use for the request | | "Headers" | {} | headers to be manually sent to the HTTP server | | "Cookies" | Automatic | cookies to pass to the server | | "StoreCookies" | True | whether to store received cookies | | "VerifyPeer" | True | verify authenticity using SSL certificates | | "UserAgent" | Automatic | user agent string to send | | "ReadTimeout" | 0 | time to allow for uploading or downloading data | | "ConnectTimeout" | 0 | time to allow for connecting to the server | | "Progress" | False | whether to raise a "progress" event | | "Transfer" | Automatic | if Automatic, the "data" event returns once and returns all the data downloaded; if "Chunks", the "data" event is raised multiple times with the data downloaded since the last event was raised | | "UserData" | None | any expression passed to this option will be stored for use of the AsynchronousTaskObject inside t ... data can be found by checking the options of AsynchronousTaskObject passed to the event function | | "FollowRedirects" | True | whether to follow redirects | * Allowed times are by default given in seconds. * Multipart elements must be of the form ``{"name", "mimetype"} -> val``, where ``val`` is a string or a list of bytes. * When an event is raised, the event function passed to ``URLFetchAsynchronous`` will be executed. The event functions will be passed three arguments, the ``AsynchronousTaskObject``, the event name, and the data received from the event. Event functions are the only method for handling data received from ``URLFetchAsynchronous``. --- ## Examples (20) ### Basic Examples (3) This opens a dialog when the download has finished: ```wl In[1]:= url = "wolfram.com"; func = Function[{asyncObj, eventType, document}, If[eventType === "data", MessageDialog["Finished downloading " <> url]]]; URLFetchAsynchronous[url, func] Out[1]= AsynchronousTaskObject["wolfram.com", 1, 28150430734831193724] ``` --- Store headers and cookies: ```wl In[1]:= headers = {}; cookies = {}; eventFunction[_, "headers", data_] := headers = data eventFunction[_, "cookies", data_] := cookies = data URLFetchAsynchronous["wolfram.com", eventFunction] Out[1]= AsynchronousTaskObject["wolfram.com", 2, 28150430734831193724] In[2]:= headers cookies Out[2]= {{"Date", "Mon, 23 Jul 2012 14:14:56 GMT"}, {"Server", "Apache"}, {"Content-Language", "en"}, {"Vary", "Accept-Language"}, {"Content-Location", "/index.en.html"}, {"Transfer-Encoding", "chunked"}, {"Content-Type", "text/html"}} Out[2]= {{"Domain" -> ".wolfram.com", "Path" -> "/", "Secure" -> "FALSE", "Expires" -> "Thu 21 Jul 2022 13:47:25", "Name" -> "WR_SID", "Value" -> "8cb1c8ef7712500d55edcb4e"}} ``` --- Use the status code and error event to determine the availability of URLs: ```wl In[1]:= statusList = {}; statusFunction[async_, "statuscode", status_] := AppendTo[statusList, {async, status}] statusFunction[async_, "error", errorStr_] := AppendTo[statusList, {async, errorStr}] URLFetchAsynchronous["http://wolfram.com", statusFunction] URLFetchAsynchronous["http://wolframalpha.com", statusFunction] URLFetchAsynchronous["http://notreal", statusFunction] Out[1]= AsynchronousTaskObject["http://wolfram.com", 3, 28150430734831193724] Out[1]= AsynchronousTaskObject["http://wolframalpha.com", 4, 28150430734831193724] Out[1]= AsynchronousTaskObject["http://notreal", 5, 28150430734831193724] In[2]:= statusList Out[2]= {{AsynchronousTaskObject["http://wolframalpha.com", 4, 28150430734831193724], {200}}, {AsynchronousTaskObject["http://wolfram.com", 3, 28150430734831193724], {200}}, {AsynchronousTaskObject["http://notreal", 5, 28150430734831193724], {"Couldn't resolve host name"}}} ``` ### Options (16) #### "Method" (1) ``"Method"`` can be used to specify the HTTP method used by the connection: ```wl In[1]:= headerData; eventFunction[_, "headers", headers_] := headerData = headers URLFetchAsynchronous["http://wolfram.com", eventFunction, "Method" -> "HEAD"] Out[1]= AsynchronousTaskObject["http://wolfram.com", 6, 28150430734831193724] In[2]:= headerData Out[2]= {{"Date", "Mon, 01 Oct 2012 20:12:20 GMT"}, {"Server", "Apache"}, {"Set-Cookie", "WR_SID=8cb1c8fe746c5069f9242948; path=/; expires=Thu, 29-Sep-22 20:12:20 GMT; domain=.wolfram.com"}, {"Content-Language", "en"}, {"Vary", "Accept-Language"}, {"Content-Location", "/index.en.html"}, {"Content-Type", "text/html"}} ``` #### "Parameters" (1) Specify the parameters that should be sent to the server: ```wl In[1]:= contentData; eventFunction[_, "data", data_] := contentData = data URLFetchAsynchronous["http://exampledata.wolfram.com/httpget.php", eventFunction, "Parameters" -> {"name" -> "Joe", "age" -> "30"}] Out[1]= AsynchronousTaskObject["http://exampledata.wolfram.com/httpget.php", 7, 28150430734831193724] In[2]:= FromCharacterCode[contentData] Out[2]= {"Joe is 30 years old. "} ``` #### "VerifyPeer" (1) Verify that the SSL certificate used by the server is valid: ```wl In[1]:= statusData; eventFunction[_, "statuscode", status_] := statusData = status; URLFetchAsynchronous["https://wolfram.com", eventFunction, "VerifyPeer" -> True] Out[1]= AsynchronousTaskObject["https://wolfram.com", 8, 28150430734831193724] In[2]:= statusData Out[2]= {200} ``` #### "Username" (1) Specify the username that should be sent to the server: ```wl In[1]:= contentData; eventFunction[_, "data", data_] := contentData = data URLFetchAsynchronous["http://exampledata.wolfram.com/authentication.php", eventFunction, "Username" -> "Joe", "Password" -> ""] Out[1]= AsynchronousTaskObject["http://exampledata.wolfram.com/authentication.php", 9, 28150430734831193724] In[2]:= FromCharacterCode[contentData] Out[2]= {"Hello Joe.You entered as your password."} ``` #### "Password" (1) Specify the password that should be sent to the server: ```wl In[1]:= contentData; eventFunction[_, "data", data_] := contentData = data URLFetchAsynchronous["http://exampledata.wolfram.com/authentication.php", eventFunction, "Username" -> "Joe", "Password" -> "topsecret"] Out[1]= AsynchronousTaskObject["http://exampledata.wolfram.com/authentication.php", 10, 28150430734831193724] In[2]:= FromCharacterCode[contentData] Out[2]= {"Hello Joe.You entered topsecret as your password."} ``` #### "UserAgent" (1) Specify the user agent that should be sent to the server: ```wl In[1]:= contentData; eventFunction[_, "data", data_] := contentData = data URLFetchAsynchronous["http://exampledata.wolfram.com/useragent.php", eventFunction, "UserAgent" -> "Web Browser"] Out[1]= AsynchronousTaskObject["http://exampledata.wolfram.com/useragent.php", 11, 28150430734831193724] In[2]:= FromCharacterCode[contentData] Out[2]= {"Web Browser"} ``` #### "Cookies" (1) Manually control the cookies used by ``URLFetchAsynchronous`` : ```wl In[1]:= contentData; eventFunction[_, "data", data_] := contentData = data URLFetchAsynchronous["http://exampledata.wolfram.com/cookies.php", eventFunction, "Cookies" -> {{"Domain" -> ".exampledata.wolfram.com", "Path" -> "/", "Secure" -> "FALSE", "Expires" -> DateString[{2020, 1, 1, 0, 0, 0}], "Name" -> "MyCookie", "Value" -> "CookieData"}}] Out[1]= AsynchronousTaskObject["http://exampledata.wolfram.com/cookies.php", 12, 28150430734831193724] In[2]:= FromCharacterCode[contentData] Out[2]= {"MyCookie--> CookieData "} ``` #### "StoreCookies" (1) If ``False``, ``"StoreCookies"`` will not place any new cookies found while connecting to the site in the global cookie share: ```wl In[1]:= cookiesData; eventFunction[_, "cookies", cookies_] := cookiesData = cookies URLFetchAsynchronous["http://wolfram.com/", eventFunction, "StoreCookies" -> False] Out[1]= AsynchronousTaskObject["http://wolfram.com/", 13, 28150430734831193724] In[2]:= cookiesData Out[2]= {{"Domain" -> ".wolfram.com", "Path" -> "/", "Secure" -> "FALSE", "Expires" -> "Thu 29 Sep 2022 20:15:58", "Name" -> "WR_SID", "Value" -> "8cb1c8fe99b5069f9feb602"}} In[3]:= $HTTPCookies Out[3]= {} ``` #### "Headers" (1) Specify headers to be sent to the server: ```wl In[1]:= contentData; eventFunction[_, "data", headers_] := contentData = headers URLFetchAsynchronous["http://exampledata.wolfram.com/headers.php", eventFunction, "Headers" -> {"MyHeader" -> "42"}] Out[1]= AsynchronousTaskObject["http://exampledata.wolfram.com/headers.php", 14, 28150430734831193724] In[2]:= FromCharacterCode[contentData] Out[2]= {" Accept: */* Host: exampledata.wolfram.com MyHeader: 42 User-Agent: Mathematica HTTPClient 9. "} ``` #### "BodyData" (1) Specify the message body that should be sent when connecting to the server: ```wl In[1]:= contentData; eventFunction[_, "data", data_] := contentData = data URLFetchAsynchronous["http://exampledata.wolfram.com/bodydata.php", eventFunction, "BodyData" -> "Message body data sent to server."] Out[1]= AsynchronousTaskObject["http://exampledata.wolfram.com/bodydata.php", 15, 28150430734831193724] In[2]:= FromCharacterCode[contentData] Out[2]= {"Message body data sent to server. "} ``` #### "MultipartData" (1) Specify the contents of a multipart message body: ```wl In[1]:= contentData; eventFunction[_, "data", data_] := contentData = data URLFetchAsynchronous["http://exampledata.wolfram.com/multipartdata.php", eventFunction, "MultipartData" -> {{"partName", "mime/type", {1, 2, 3, 4, 5}}}] Out[1]= AsynchronousTaskObject["http://exampledata.wolfram.com/multipartdata.php", 16, 28150430734831193724] In[2]:= FromCharacterCode[contentData] Out[2]= {"7cfdd07889b3295d6a550914ab35e068"} ``` #### "ReadTimeout" (1) Specify the maximum allotted time in seconds to finish downloading data from the server: ```wl In[1]:= errorString; eventFunction[_, "error", error_] := errorString = error URLFetchAsynchronous["http://exampledata.wolfram.com/50mb.dat", eventFunction, "ReadTimeout" -> 1] Out[1]= AsynchronousTaskObject["http://exampledata.wolfram.com/50mb.dat", 17, 28150430734831193724] In[2]:= errorString Out[2]= {"Timeout was reached"} ``` #### "ConnectTimeout" (1) Specify the maximum allotted time in seconds to establish a connection to the server: ```wl In[1]:= errorString; eventFunction[_, "error", error_] := errorString = error URLFetchAsynchronous["http://127.0.0.1", eventFunction, "ConnectTimeout" -> 1] Out[1]= AsynchronousTaskObject["http://127.0.0.1", 18, 28150430734831193724] In[2]:= errorString Out[2]= {"Timeout was reached"} ``` #### "Progress" (1) If ``True``, the ``"progress"`` event will be raised, returning information downloaded and uploaded about in-progress downloads: ```wl In[1]:= progressData = {}; In[2]:= progressFunction[_, "progress", progress_] := progressData = progress URLFetchAsynchronous["http://exampledata.wolfram.com/5mb.dat", progressFunction, "Progress" -> True]; Dynamic[progressData] Out[2]= Dynamic[progressData] ``` #### "Transfer" (1) If ``"Transfer"`` is set to ``"Chunks"``, data will be returned as soon as it is available from the server, instead of waiting for the entire connection to finish: ```wl In[1]:= chunks = {}; In[2]:= chunkFunction[_, "data", data_] := AppendTo[chunks, data] URLFetchAsynchronous["http://wolfram.com", chunkFunction, "Transfer" -> "Chunks"]; In[3]:= Length[chunks] Out[3]= 26 ``` #### "UserData" (1) ``"UserData"`` is any expression containing information to be made available when an event is called: ```wl In[1]:= dataString; eventFunction[async_, "data", data_] := dataString = "UserData" /. Options[async] URLFetchAsynchronous["http://wolfram.com", eventFunction, "UserData" -> "stored data"]; In[2]:= dataString Out[2]= "stored data" ``` ### Properties & Relations (1) Use ``URLSaveAsynchronous`` to store output in a file: ```wl In[1]:= f[_, "statuscode", _] := CreateDialog[{"Finished download", DefaultButton[]}] URLSaveAsynchronous["http://wolfram.com", FileNameJoin[{$TemporaryDirectory, "index.html"}], f] Out[1]= AsynchronousTaskObject["http://wolfram.com", 20, 28150430734831193724] ``` ## History * [Introduced in 2012 (9.0)](https://reference.wolfram.com/language/guide/SummaryOfNewFeaturesIn90.en.md)